2023-10-21T03:51:41.515 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-21T03:51:41.516 DEBUG:teuthology.run:Teuthology command: teuthology --description rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/ec features/layering msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} --owner scheduled_rraja@teuthology --archive /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433369 --name rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi --verbose -- /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433369/orig.config.yaml 2023-10-21T03:51:41.548 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T03:51:41.633 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433369 branch: ajarr-wip-extra-debug-62994-oct-19 description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/ec features/layering msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7433369' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: ajarr-wip-extra-debug-62994-oct-19 ceph: conf: client: rbd default data pool: datapool rbd default features: 1 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 enable experimental unrecoverable data corrupting features: '*' mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug randomize hobject sort order: false osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: default sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: ajarr-wip-extra-debug-62994-oct-19 sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab owner: scheduled_rraja@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1449 sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab sleep_before_teardown: 0 suite: rbd:cli suite_branch: ajarr-wip-extra-debug-62994-oct-19 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab targets: smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPC+zfLfb5HR102ZeiDb68RvA1E5FrxGqtkVP/oLWYY5xf6lz9rSaD7Wy5qIbUtrDsS7z0KZ/yJdQAhgMzPfMtY= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd erasure-code-profile set teuthologyprofile crush-failure-domain=osd m=1 k=2 - sudo ceph osd pool create datapool 4 4 erasure teuthologyprofile - sudo ceph osd pool set datapool allow_ec_overwrites true - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-20_15:15:00 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-21T03:51:41.633 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa; will attempt to use it 2023-10-21T03:51:41.634 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks 2023-10-21T03:51:41.635 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-21T03:51:41.636 INFO:teuthology.task.internal:Checking packages... 2023-10-21T03:51:41.661 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab' 2023-10-21T03:51:41.662 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-21T03:51:41.662 INFO:teuthology.packaging:ref: None 2023-10-21T03:51:41.662 INFO:teuthology.packaging:tag: None 2023-10-21T03:51:41.663 INFO:teuthology.packaging:branch: ajarr-wip-extra-debug-62994-oct-19 2023-10-21T03:51:41.663 INFO:teuthology.packaging:sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T03:51:41.663 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=ajarr-wip-extra-debug-62994-oct-19 2023-10-21T03:51:41.810 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T03:51:41.810 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-21T03:51:41.816 INFO:teuthology.task.internal:no buildpackages task found 2023-10-21T03:51:41.817 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-21T03:51:41.823 INFO:teuthology.task.internal:Saving configuration 2023-10-21T03:51:41.838 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-21T03:51:41.844 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-21T03:51:41.871 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433369', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-21 03:45:45.604199', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILse8fwLAnhb1nBAmujJuGWOKbqxTVyezNQpDliqcec+'} 2023-10-21T03:51:41.871 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-21T03:51:41.877 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-21T03:51:41.877 INFO:teuthology.run_tasks:Running task console_log... 2023-10-21T03:51:41.920 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2842828af0>, signals=[15]) 2023-10-21T03:51:41.921 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-21T03:51:41.927 INFO:teuthology.task.internal:Opening connections... 2023-10-21T03:51:41.928 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2023-10-21T03:51:41.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:51:42.003 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-21T03:51:42.009 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2023-10-21T03:51:42.115 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2023-10-21T03:51:42.115 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2023-10-21T03:51:42.162 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-21T03:51:42.163 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Ubuntu" 2023-10-21T03:51:42.163 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="22.04" 2023-10-21T03:51:42.163 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-21T03:51:42.163 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_CODENAME=jammy 2023-10-21T03:51:42.163 INFO:teuthology.orchestra.run.smithi172.stdout:ID=ubuntu 2023-10-21T03:51:42.164 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE=debian 2023-10-21T03:51:42.164 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-21T03:51:42.164 INFO:teuthology.orchestra.run.smithi172.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-21T03:51:42.164 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-21T03:51:42.164 INFO:teuthology.orchestra.run.smithi172.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-21T03:51:42.165 INFO:teuthology.orchestra.run.smithi172.stdout:UBUNTU_CODENAME=jammy 2023-10-21T03:51:42.165 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2023-10-21T03:51:42.185 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-21T03:51:42.194 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-21T03:51:42.199 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-21T03:51:42.200 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2023-10-21T03:51:42.210 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-21T03:51:42.217 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-21T03:51:42.218 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2023-10-21T03:51:42.260 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-21T03:51:42.293 INFO:teuthology.run_tasks:Running task kernel... 2023-10-21T03:51:42.309 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-21T03:51:42.310 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-21T03:51:42.310 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-21T03:51:42.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T03:51:42.317 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2023-10-21T03:51:42.367 INFO:teuthology.orchestra.run.smithi172.stdout:5.15.0-56-generic 2023-10-21T03:51:42.367 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.15.0-56-generic 2023-10-21T03:51:42.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2023-10-21T03:51:42.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2023-10-21T03:51:42.642 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-21T03:51:42.754 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-21T03:51:42.836 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-21T03:51:42.852 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [896 kB] 2023-10-21T03:51:42.950 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [353 kB] 2023-10-21T03:51:42.956 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [180 kB] 2023-10-21T03:51:42.958 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-21T03:51:42.959 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,016 kB] 2023-10-21T03:51:42.979 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-21T03:51:42.980 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [164 kB] 2023-10-21T03:51:42.982 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-10-21T03:51:42.982 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-10-21T03:51:42.990 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [793 kB] 2023-10-21T03:51:43.001 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-21T03:51:43.002 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-10-21T03:51:43.003 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-21T03:51:43.003 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-21T03:51:43.037 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-21T03:51:43.038 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-21T03:51:43.190 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-21T03:51:43.329 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [515 kB] 2023-10-21T03:51:43.513 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,103 kB] 2023-10-21T03:51:43.641 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [239 kB] 2023-10-21T03:51:43.657 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-21T03:51:43.659 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,036 kB] 2023-10-21T03:51:43.711 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-10-21T03:51:43.712 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [167 kB] 2023-10-21T03:51:43.719 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-10-21T03:51:43.883 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-10-21T03:51:44.453 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [995 kB] 2023-10-21T03:51:44.563 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-10-21T03:51:44.582 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-21T03:51:44.584 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-21T03:51:44.587 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-21T03:51:44.588 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-21T03:51:44.588 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-21T03:51:44.589 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-10-21T03:51:44.592 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-10-21T03:51:44.597 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-21T03:51:44.762 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-21T03:51:44.854 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-10-21T03:51:44.955 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-10-21T03:51:44.990 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-21T03:51:45.052 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-10-21T03:51:49.575 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 9,805 kB in 2s (3,953 kB/s) 2023-10-21T03:51:50.851 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-10-21T03:51:50.868 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-21T03:51:50.921 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-10-21T03:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2023-10-21T03:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2023-10-21T03:51:51.225 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T03:51:51.226 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-21T03:51:51.226 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-21T03:51:51.226 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-21T03:51:51.227 INFO:teuthology.orchestra.run.smithi172.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-21T03:51:51.227 INFO:teuthology.orchestra.run.smithi172.stdout: python2.7-minimal 2023-10-21T03:51:51.227 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T03:51:51.228 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2023-10-21T03:51:51.228 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-21T03:51:51.228 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-generic linux-image-5.15.0-87-generic 2023-10-21T03:51:51.228 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-5.15.0-87-generic linux-modules-extra-5.15.0-87-generic 2023-10-21T03:51:51.229 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2023-10-21T03:51:51.229 INFO:teuthology.orchestra.run.smithi172.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-21T03:51:51.229 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2023-10-21T03:51:51.229 INFO:teuthology.orchestra.run.smithi172.stdout: thermald 2023-10-21T03:51:51.271 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2023-10-21T03:51:51.271 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-21T03:51:51.271 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.15.0-87-generic linux-modules-5.15.0-87-generic 2023-10-21T03:51:51.272 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-extra-5.15.0-87-generic 2023-10-21T03:51:51.272 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages will be upgraded: 2023-10-21T03:51:51.273 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-21T03:51:51.450 INFO:teuthology.orchestra.run.smithi172.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-10-21T03:51:51.451 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 113 MB of archives. 2023-10-21T03:51:51.451 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-21T03:51:51.451 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-87-generic amd64 5.15.0-87.97 [22.4 MB] 2023-10-21T03:51:52.946 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-87-generic amd64 5.15.0-87.97 [11.5 MB] 2023-10-21T03:51:53.294 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-87-generic amd64 5.15.0-87.97 [64.0 MB] 2023-10-21T03:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.87.84 [1,698 B] 2023-10-21T03:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.87.84 [2,482 B] 2023-10-21T03:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87 all 5.15.0-87.97 [12.3 MB] 2023-10-21T03:51:56.410 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87-generic amd64 5.15.0-87.97 [2,888 kB] 2023-10-21T03:51:56.655 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.87.84 [2,322 B] 2023-10-21T03:51:57.050 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 113 MB in 5s (21.0 MB/s) 2023-10-21T03:51:57.203 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-5.15.0-87-generic. 2023-10-21T03:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-21T03:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../0-linux-modules-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-21T03:51:59.274 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:01.441 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-image-5.15.0-87-generic. 2023-10-21T03:52:01.462 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../1-linux-image-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-21T03:52:01.551 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:02.036 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-87-generic. 2023-10-21T03:52:02.057 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-21T03:52:02.094 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:08.468 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../3-linux-generic_5.15.0.87.84_amd64.deb ... 2023-10-21T03:52:08.593 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-21T03:52:08.942 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.87.84_amd64.deb ... 2023-10-21T03:52:09.063 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-21T03:52:09.351 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.15.0-87. 2023-10-21T03:52:09.374 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../5-linux-headers-5.15.0-87_5.15.0-87.97_all.deb ... 2023-10-21T03:52:09.417 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-21T03:52:15.150 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.15.0-87-generic. 2023-10-21T03:52:15.170 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../6-linux-headers-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-21T03:52:15.209 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:17.760 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.87.84_amd64.deb ... 2023-10-21T03:52:17.885 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-21T03:52:18.786 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-21T03:52:18.915 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:19.041 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-generic (5.15.0.87.84) ... 2023-10-21T03:52:19.168 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:20.587 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:21.921 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-21T03:52:21.922 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-21T03:52:21.922 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-87-generic 2023-10-21T03:52:21.922 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-87-generic 2023-10-21T03:52:22.039 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:23.399 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-generic (5.15.0.87.84) ... 2023-10-21T03:52:23.549 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-generic (5.15.0.87.84) ... 2023-10-21T03:52:23.676 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:23.732 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-21T03:52:23.732 INFO:teuthology.orchestra.run.smithi172.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-87-generic 2023-10-21T03:52:37.324 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-21T03:52:37.325 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub' 2023-10-21T03:52:37.334 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-21T03:52:37.359 INFO:teuthology.orchestra.run.smithi172.stdout:Generating grub configuration file ... 2023-10-21T03:52:37.635 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-21T03:52:37.662 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-21T03:52:37.940 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-21T03:52:37.946 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-21T03:52:38.019 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-21T03:52:38.024 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-21T03:52:38.202 INFO:teuthology.orchestra.run.smithi172.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-21T03:52:38.202 INFO:teuthology.orchestra.run.smithi172.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-21T03:52:38.203 INFO:teuthology.orchestra.run.smithi172.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-21T03:52:38.247 INFO:teuthology.orchestra.run.smithi172.stdout:done 2023-10-21T03:52:38.709 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2023-10-21T03:52:38.731 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2023-10-21T03:52:38.731 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2023-10-21T03:52:38.732 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2023-10-21T03:52:38.732 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2023-10-21T03:52:38.732 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 20 2023-10-21T03:52:38.732 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2023-10-21T03:52:38.732 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2023-10-21T03:52:38.733 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2023-10-21T03:52:38.733 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.15.0.87.84 2023-10-21T03:52:38.733 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-21T03:52:38.733 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-21T03:52:38.734 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2023-10-21T03:52:38.734 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2023-10-21T03:52:38.734 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2023-10-21T03:52:38.734 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2023-10-21T03:52:38.734 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-87-generic 2023-10-21T03:52:38.735 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-21T03:52:38.735 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-21T03:52:38.735 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-21T03:52:38.735 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2023-10-21T03:52:38.736 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2023-10-21T03:52:38.889 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2023-10-21T03:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-21T03:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-21T03:52:39.207 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-21T03:52:39.289 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-21T03:52:40.433 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-10-21T03:52:40.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-21T03:52:40.502 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-10-21T03:52:40.666 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2023-10-21T03:52:40.666 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2023-10-21T03:52:40.811 INFO:teuthology.orchestra.run.smithi172.stdout:linux-image-generic is already the newest version (5.15.0.87.84). 2023-10-21T03:52:40.811 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T03:52:40.811 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-21T03:52:40.812 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-21T03:52:40.812 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-21T03:52:40.812 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-21T03:52:40.812 INFO:teuthology.orchestra.run.smithi172.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-21T03:52:40.812 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T03:52:40.847 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-10-21T03:52:40.849 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2023-10-21T03:52:40.904 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2023-10-21T03:52:40.904 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2023-10-21T03:52:40.904 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2023-10-21T03:52:40.905 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2023-10-21T03:52:40.905 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 20 2023-10-21T03:52:40.905 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2023-10-21T03:52:40.905 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2023-10-21T03:52:40.905 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2023-10-21T03:52:40.906 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.15.0.87.84 2023-10-21T03:52:40.906 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-21T03:52:40.906 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-21T03:52:40.906 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2023-10-21T03:52:40.907 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2023-10-21T03:52:40.907 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2023-10-21T03:52:40.907 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2023-10-21T03:52:40.907 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2023-10-21T03:52:40.949 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.idSelNALwe 2023-10-21T03:52:40.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub/grub.cfg /tmp/tmp.idSelNALwe 2023-10-21T03:52:41.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.idSelNALwe 2023-10-21T03:52:41.133 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.idSelNALwe is 10KB 2023-10-21T03:52:41.147 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.idSelNALwe 2023-10-21T03:52:41.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-21T03:52:41.205 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T03:52:41.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-21T03:52:41.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-21T03:52:41.272 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-87-generic 2023-10-21T03:52:41.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo update-grub 2023-10-21T03:52:42.381 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub' 2023-10-21T03:52:42.383 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-21T03:52:42.385 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2023-10-21T03:52:42.557 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-21T03:52:42.563 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-21T03:52:42.781 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-21T03:52:42.786 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-21T03:52:42.829 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-21T03:52:42.834 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-21T03:52:42.950 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-21T03:52:42.950 INFO:teuthology.orchestra.run.smithi172.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-21T03:52:42.951 INFO:teuthology.orchestra.run.smithi172.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-21T03:52:42.962 INFO:teuthology.orchestra.run.smithi172.stderr:done 2023-10-21T03:52:42.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2023-10-21T03:53:13.010 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-21T03:53:13.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-21T03:53:13.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:53:31.514 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-10-21T03:53:40.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-21T03:53:40.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:53:43.578 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-10-21T03:53:55.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-21T03:53:55.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:54:55.650 DEBUG:teuthology.orchestra.remote:timed out 2023-10-21T03:55:10.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-21T03:55:10.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:55:10.843 DEBUG:teuthology.orchestra.run.smithi172:> true 2023-10-21T03:55:11.378 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-21T03:55:11.378 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-87-generic"... 2023-10-21T03:55:11.378 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2023-10-21T03:55:11.424 INFO:teuthology.orchestra.run.smithi172.stdout:5.15.0-87-generic 2023-10-21T03:55:11.424 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-87-generic vs 5.15.0-87-generic 2023-10-21T03:55:11.425 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-21T03:55:11.425 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-21T03:55:12.426 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-21T03:55:12.427 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-21T03:55:12.510 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2023-10-21T03:55:12.530 DEBUG:teuthology.parallel:result is None 2023-10-21T03:55:12.530 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-21T03:55:12.538 INFO:teuthology.task.internal:Creating test directory... 2023-10-21T03:55:12.539 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-21T03:55:12.544 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-21T03:55:12.551 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-21T03:55:12.561 INFO:teuthology.task.internal:Creating archive directory... 2023-10-21T03:55:12.561 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-21T03:55:12.615 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-21T03:55:12.625 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-21T03:55:12.626 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-21T03:55:12.674 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-21T03:55:12.681 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-21T03:55:12.683 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-21T03:55:12.690 INFO:teuthology.task.internal:Configuring sudo... 2023-10-21T03:55:12.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-21T03:55:12.736 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-21T03:55:12.746 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-21T03:55:12.746 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-21T03:55:12.784 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-21T03:55:12.832 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-21T03:55:12.881 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T03:55:12.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-21T03:55:12.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2023-10-21T03:55:13.051 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-21T03:55:13.058 INFO:teuthology.task.internal:Starting timer... 2023-10-21T03:55:13.059 INFO:teuthology.run_tasks:Running task pcp... 2023-10-21T03:55:13.071 INFO:teuthology.run_tasks:Running task selinux... 2023-10-21T03:55:13.080 DEBUG:teuthology.task.selinux:Excluding smithi172: OS 'ubuntu' does not support SELinux 2023-10-21T03:55:13.080 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-21T03:55:13.080 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-21T03:55:13.081 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-21T03:55:13.081 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-21T03:55:13.091 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-21T03:55:13.097 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-21T03:55:13.209 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-21T03:55:13.228 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-21T03:55:13.230 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-21T04:02:56.403 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2023-10-21T04:02:56.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-21T04:02:56.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T04:02:56.481 DEBUG:teuthology.orchestra.run.smithi172:> true 2023-10-21T04:02:56.574 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-21T04:02:56.575 INFO:teuthology.run_tasks:Running task clock... 2023-10-21T04:02:56.587 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-21T04:02:56.588 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-21T04:02:56.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-21T04:02:56.642 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-21T04:02:56.643 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: Command line: ntpd -gq 2023-10-21T04:02:56.643 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: ---------------------------------------------------- 2023-10-21T04:02:56.643 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: ntp-4 is maintained by Network Time Foundation, 2023-10-21T04:02:56.643 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-21T04:02:56.644 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: corporation. Support and training for ntp-4 are 2023-10-21T04:02:56.644 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: available at https://www.nwtime.org/support 2023-10-21T04:02:56.644 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: ---------------------------------------------------- 2023-10-21T04:02:56.645 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: proto: precision = 0.058 usec (-24) 2023-10-21T04:02:56.645 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: basedate set to 2022-02-04 2023-10-21T04:02:56.645 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: gps base set to 2022-02-06 (week 2196) 2023-10-21T04:02:56.646 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-21T04:02:56.646 INFO:teuthology.orchestra.run.smithi172.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-21T04:02:56.646 INFO:teuthology.orchestra.run.smithi172.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-21T04:02:56.647 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: restrict ::: KOD does nothing without LIMITED. 2023-10-21T04:02:56.647 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: Listen and drop on 0 v6wildcard [::]:123 2023-10-21T04:02:56.648 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-21T04:02:56.648 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: Listen normally on 2 lo 127.0.0.1:123 2023-10-21T04:02:56.648 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: Listen normally on 3 enp3s0f1 172.21.15.172:123 2023-10-21T04:02:56.648 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: Listen normally on 4 lo [::1]:123 2023-10-21T04:02:56.648 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f91%5]:123 2023-10-21T04:02:56.649 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:02:56 ntpd[27978]: Listening on routing socket on fd #22 for interface updates 2023-10-21T04:02:57.643 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:02:57 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:57.643 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:02:57 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:00.643 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:00 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:00.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:00 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:00.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:00 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:00.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:00 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:00.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:00 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:03:00.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:00 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:03:01.646 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:01 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:01.647 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:01 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:01.647 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:01 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:01.647 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:01 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:01.648 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:01 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:01.649 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:01 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:01.649 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:01 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:03:01.649 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:01 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:03:01.649 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:01 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:01.650 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:01 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:01.650 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:01 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:03:01.650 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:01 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:03:02.643 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:02 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:02.643 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:02 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:02.643 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:02 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:02.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:02 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:02.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:02 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:03:02.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:02 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:03:03.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:03 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:03.645 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:03 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:03.645 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:03 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:03.645 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:03 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:03.645 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:03 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:03.646 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:03 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:03.646 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:03 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:03:03.646 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:03 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:03:03.646 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:03 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:03.647 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:03 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:03.647 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:03 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:03:03.647 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:03 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:03:04.643 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:04 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:04.643 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:04 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:04.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:04 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:04.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:04 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:04.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:04 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:03:04.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:04 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:03:05.643 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:05 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:05.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:05 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:05.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:05 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:05.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:05 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:05.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:05 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:05.645 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:05 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:05.645 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:05 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:03:05.645 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:05 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:03:05.645 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:05 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:05.645 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:05 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:05 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:03:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:05 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:03:06.643 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:06 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:06.643 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:06 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:06.643 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:06 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:03:06.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:06 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:03:06.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:06 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:03:06.644 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:06 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:03:06.645 INFO:teuthology.orchestra.run.smithi172.stdout:21 Oct 04:03:06 ntpd[27978]: ntpd: time slew -0.002476 s 2023-10-21T04:03:06.645 INFO:teuthology.orchestra.run.smithi172.stdout:ntpd: time slew -0.002476s 2023-10-21T04:03:06.645 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:06 ntpd[27978]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-21T04:03:06.645 INFO:teuthology.orchestra.run.smithi172.stderr:21 Oct 04:03:06 ntpd[27978]: can't open /var/log/ntpstats/loopstats.20231021: Permission denied 2023-10-21T04:03:06.731 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2023-10-21T04:03:06.732 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2023-10-21T04:03:06.732 INFO:teuthology.orchestra.run.smithi172.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-21T04:03:06.732 INFO:teuthology.orchestra.run.smithi172.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-21T04:03:06.732 INFO:teuthology.orchestra.run.smithi172.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-21T04:03:06.733 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-21T04:03:06.734 INFO:teuthology.run_tasks:Running task install... 2023-10-21T04:03:06.743 DEBUG:teuthology.task.install:project ceph 2023-10-21T04:03:06.743 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}} 2023-10-21T04:03:06.743 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'} 2023-10-21T04:03:06.744 INFO:teuthology.task.install:Using flavor: default 2023-10-21T04:03:06.749 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-21T04:03:06.750 INFO:teuthology.task.install:extra packages: [] 2023-10-21T04:03:06.750 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-key list | grep Ceph 2023-10-21T04:03:06.830 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-21T04:03:06.856 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-21T04:03:06.857 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph.com (release key) 2023-10-21T04:03:06.857 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-21T04:03:06.858 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:03:07.004 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default/ 2023-10-21T04:03:07.004 INFO:teuthology.task.install.deb:Package version is 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:03:07.197 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:03:07.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-21T04:03:07.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2023-10-21T04:03:07.375 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-21T04:03:07.384 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-21T04:03:07.401 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-21T04:03:07.436 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy InRelease 2023-10-21T04:03:07.458 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2023-10-21T04:03:07.480 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-21T04:03:07.498 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-21T04:03:07.501 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2023-10-21T04:03:07.529 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-21T04:03:07.731 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 27.9 kB in 0s (81.0 kB/s) 2023-10-21T04:03:08.575 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-10-21T04:03:08.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6744-g7f103c7c-1jammy cephadm=18.0.0-6744-g7f103c7c-1jammy ceph-mds=18.0.0-6744-g7f103c7c-1jammy ceph-mgr=18.0.0-6744-g7f103c7c-1jammy ceph-common=18.0.0-6744-g7f103c7c-1jammy ceph-fuse=18.0.0-6744-g7f103c7c-1jammy ceph-test=18.0.0-6744-g7f103c7c-1jammy ceph-volume=18.0.0-6744-g7f103c7c-1jammy radosgw=18.0.0-6744-g7f103c7c-1jammy python3-rados=18.0.0-6744-g7f103c7c-1jammy python3-rgw=18.0.0-6744-g7f103c7c-1jammy python3-cephfs=18.0.0-6744-g7f103c7c-1jammy python3-rbd=18.0.0-6744-g7f103c7c-1jammy libcephfs2=18.0.0-6744-g7f103c7c-1jammy libcephfs-dev=18.0.0-6744-g7f103c7c-1jammy librados2=18.0.0-6744-g7f103c7c-1jammy librbd1=18.0.0-6744-g7f103c7c-1jammy rbd-fuse=18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:03:08.641 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-10-21T04:03:08.805 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2023-10-21T04:03:08.806 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2023-10-21T04:03:08.954 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T04:03:08.954 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-21T04:03:08.954 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-21T04:03:08.954 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-21T04:03:08.955 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-21T04:03:08.956 INFO:teuthology.orchestra.run.smithi172.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-21T04:03:08.956 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T04:03:08.956 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2023-10-21T04:03:08.956 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-21T04:03:08.957 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-21T04:03:08.957 INFO:teuthology.orchestra.run.smithi172.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-21T04:03:08.957 INFO:teuthology.orchestra.run.smithi172.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-21T04:03:08.957 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-21T04:03:08.957 INFO:teuthology.orchestra.run.smithi172.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-21T04:03:08.958 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-21T04:03:08.958 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-21T04:03:08.958 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-21T04:03:08.958 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-21T04:03:08.958 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-21T04:03:08.959 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-21T04:03:08.959 INFO:teuthology.orchestra.run.smithi172.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-21T04:03:08.959 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-21T04:03:08.959 INFO:teuthology.orchestra.run.smithi172.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-21T04:03:08.959 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-21T04:03:08.959 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-21T04:03:08.960 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-21T04:03:08.960 INFO:teuthology.orchestra.run.smithi172.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-21T04:03:08.960 INFO:teuthology.orchestra.run.smithi172.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-21T04:03:08.960 INFO:teuthology.orchestra.run.smithi172.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-21T04:03:08.960 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-21T04:03:08.961 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2023-10-21T04:03:08.961 INFO:teuthology.orchestra.run.smithi172.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-21T04:03:08.961 INFO:teuthology.orchestra.run.smithi172.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-21T04:03:08.962 INFO:teuthology.orchestra.run.smithi172.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-21T04:03:08.962 INFO:teuthology.orchestra.run.smithi172.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-21T04:03:08.962 INFO:teuthology.orchestra.run.smithi172.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-21T04:03:08.962 INFO:teuthology.orchestra.run.smithi172.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-21T04:03:08.962 INFO:teuthology.orchestra.run.smithi172.stdout: python3-watchdog 2023-10-21T04:03:08.962 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2023-10-21T04:03:08.963 INFO:teuthology.orchestra.run.smithi172.stdout: btrfs-tools 2023-10-21T04:03:09.051 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2023-10-21T04:03:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-21T04:03:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-21T04:03:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-21T04:03:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-21T04:03:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-21T04:03:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-21T04:03:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-21T04:03:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-21T04:03:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-21T04:03:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-21T04:03:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-21T04:03:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-21T04:03:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-21T04:03:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-21T04:03:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-21T04:03:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-21T04:03:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-21T04:03:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-21T04:03:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-21T04:03:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-21T04:03:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-21T04:03:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-21T04:03:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-21T04:03:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-21T04:03:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: socat xmlstarlet 2023-10-21T04:03:09.130 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 109 newly installed, 0 to remove and 257 not upgraded. 2023-10-21T04:03:09.131 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 184 MB of archives. 2023-10-21T04:03:09.131 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 824 MB of additional disk space will be used. 2023-10-21T04:03:09.131 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-21T04:03:09.187 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-21T04:03:09.223 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-21T04:03:09.225 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-21T04:03:09.242 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-21T04:03:09.278 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6744-g7f103c7c-1jammy [3,450 kB] 2023-10-21T04:03:09.292 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-21T04:03:09.296 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-21T04:03:09.308 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-21T04:03:09.312 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-21T04:03:09.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-21T04:03:09.314 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-21T04:03:09.322 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-21T04:03:09.323 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-21T04:03:09.323 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-21T04:03:09.324 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-21T04:03:09.330 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-21T04:03:09.330 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-21T04:03:09.346 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-21T04:03:09.347 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-21T04:03:09.347 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-21T04:03:09.348 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-21T04:03:09.350 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-21T04:03:09.351 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-21T04:03:09.351 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-21T04:03:09.351 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-21T04:03:09.363 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-21T04:03:09.363 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-21T04:03:09.380 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-21T04:03:09.381 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-21T04:03:09.381 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-21T04:03:09.382 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-21T04:03:09.387 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-21T04:03:09.388 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-21T04:03:09.388 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-21T04:03:09.389 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-21T04:03:09.397 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-21T04:03:09.398 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-21T04:03:09.408 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-21T04:03:09.409 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-21T04:03:09.410 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-21T04:03:09.412 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-21T04:03:09.413 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-21T04:03:09.415 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-21T04:03:09.467 INFO:teuthology.orchestra.run.smithi172.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-21T04:03:09.468 INFO:teuthology.orchestra.run.smithi172.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-21T04:03:09.469 INFO:teuthology.orchestra.run.smithi172.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-21T04:03:09.481 INFO:teuthology.orchestra.run.smithi172.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-21T04:03:09.481 INFO:teuthology.orchestra.run.smithi172.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-21T04:03:09.482 INFO:teuthology.orchestra.run.smithi172.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-21T04:03:09.500 INFO:teuthology.orchestra.run.smithi172.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-21T04:03:09.501 INFO:teuthology.orchestra.run.smithi172.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-21T04:03:09.502 INFO:teuthology.orchestra.run.smithi172.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-21T04:03:09.518 INFO:teuthology.orchestra.run.smithi172.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-21T04:03:09.518 INFO:teuthology.orchestra.run.smithi172.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-21T04:03:09.519 INFO:teuthology.orchestra.run.smithi172.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-21T04:03:09.519 INFO:teuthology.orchestra.run.smithi172.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-21T04:03:09.520 INFO:teuthology.orchestra.run.smithi172.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-21T04:03:09.521 INFO:teuthology.orchestra.run.smithi172.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-21T04:03:09.521 INFO:teuthology.orchestra.run.smithi172.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-21T04:03:09.535 INFO:teuthology.orchestra.run.smithi172.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-21T04:03:09.538 INFO:teuthology.orchestra.run.smithi172.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-21T04:03:09.540 INFO:teuthology.orchestra.run.smithi172.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-21T04:03:09.552 INFO:teuthology.orchestra.run.smithi172.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-21T04:03:09.553 INFO:teuthology.orchestra.run.smithi172.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-21T04:03:09.556 INFO:teuthology.orchestra.run.smithi172.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-21T04:03:09.558 INFO:teuthology.orchestra.run.smithi172.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-21T04:03:09.562 INFO:teuthology.orchestra.run.smithi172.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-21T04:03:09.565 INFO:teuthology.orchestra.run.smithi172.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-21T04:03:09.565 INFO:teuthology.orchestra.run.smithi172.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-21T04:03:09.569 INFO:teuthology.orchestra.run.smithi172.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-21T04:03:09.569 INFO:teuthology.orchestra.run.smithi172.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-21T04:03:09.570 INFO:teuthology.orchestra.run.smithi172.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-21T04:03:09.580 INFO:teuthology.orchestra.run.smithi172.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6744-g7f103c7c-1jammy [3,132 kB] 2023-10-21T04:03:09.585 INFO:teuthology.orchestra.run.smithi172.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-21T04:03:09.592 INFO:teuthology.orchestra.run.smithi172.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-21T04:03:09.592 INFO:teuthology.orchestra.run.smithi172.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-21T04:03:09.592 INFO:teuthology.orchestra.run.smithi172.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-21T04:03:09.594 INFO:teuthology.orchestra.run.smithi172.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-21T04:03:09.595 INFO:teuthology.orchestra.run.smithi172.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-21T04:03:09.675 INFO:teuthology.orchestra.run.smithi172.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6744-g7f103c7c-1jammy [689 kB] 2023-10-21T04:03:09.697 INFO:teuthology.orchestra.run.smithi172.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6744-g7f103c7c-1jammy [356 kB] 2023-10-21T04:03:09.709 INFO:teuthology.orchestra.run.smithi172.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6744-g7f103c7c-1jammy [32.8 kB] 2023-10-21T04:03:09.710 INFO:teuthology.orchestra.run.smithi172.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6744-g7f103c7c-1jammy [180 kB] 2023-10-21T04:03:09.716 INFO:teuthology.orchestra.run.smithi172.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6744-g7f103c7c-1jammy [62.4 kB] 2023-10-21T04:03:09.718 INFO:teuthology.orchestra.run.smithi172.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6744-g7f103c7c-1jammy [330 kB] 2023-10-21T04:03:09.729 INFO:teuthology.orchestra.run.smithi172.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6744-g7f103c7c-1jammy [6,221 kB] 2023-10-21T04:03:09.934 INFO:teuthology.orchestra.run.smithi172.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6744-g7f103c7c-1jammy [112 kB] 2023-10-21T04:03:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6744-g7f103c7c-1jammy [441 kB] 2023-10-21T04:03:09.951 INFO:teuthology.orchestra.run.smithi172.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6744-g7f103c7c-1jammy [22.6 MB] 2023-10-21T04:03:10.848 INFO:teuthology.orchestra.run.smithi172.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6744-g7f103c7c-1jammy [6,852 kB] 2023-10-21T04:03:10.985 INFO:teuthology.orchestra.run.smithi172.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6744-g7f103c7c-1jammy [238 kB] 2023-10-21T04:03:10.991 INFO:teuthology.orchestra.run.smithi172.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6744-g7f103c7c-1jammy [128 kB] 2023-10-21T04:03:10.995 INFO:teuthology.orchestra.run.smithi172.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6744-g7f103c7c-1jammy [1,496 kB] 2023-10-21T04:03:11.045 INFO:teuthology.orchestra.run.smithi172.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6744-g7f103c7c-1jammy [9,863 kB] 2023-10-21T04:03:11.370 INFO:teuthology.orchestra.run.smithi172.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6744-g7f103c7c-1jammy [31.7 MB] 2023-10-21T04:03:12.551 INFO:teuthology.orchestra.run.smithi172.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6744-g7f103c7c-1jammy [14.0 kB] 2023-10-21T04:03:12.551 INFO:teuthology.orchestra.run.smithi172.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6744-g7f103c7c-1jammy [839 kB] 2023-10-21T04:03:12.568 INFO:teuthology.orchestra.run.smithi172.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6744-g7f103c7c-1jammy [2,146 kB] 2023-10-21T04:03:12.605 INFO:teuthology.orchestra.run.smithi172.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6744-g7f103c7c-1jammy [283 kB] 2023-10-21T04:03:12.609 INFO:teuthology.orchestra.run.smithi172.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6744-g7f103c7c-1jammy [138 kB] 2023-10-21T04:03:12.611 INFO:teuthology.orchestra.run.smithi172.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6744-g7f103c7c-1jammy [1,906 kB] 2023-10-21T04:03:12.633 INFO:teuthology.orchestra.run.smithi172.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6744-g7f103c7c-1jammy [8,625 kB] 2023-10-21T04:03:12.913 INFO:teuthology.orchestra.run.smithi172.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6744-g7f103c7c-1jammy [14.2 kB] 2023-10-21T04:03:12.914 INFO:teuthology.orchestra.run.smithi172.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6744-g7f103c7c-1jammy [47.8 MB] 2023-10-21T04:03:14.742 INFO:teuthology.orchestra.run.smithi172.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6744-g7f103c7c-1jammy [120 kB] 2023-10-21T04:03:14.745 INFO:teuthology.orchestra.run.smithi172.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6744-g7f103c7c-1jammy [38.3 kB] 2023-10-21T04:03:14.746 INFO:teuthology.orchestra.run.smithi172.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6744-g7f103c7c-1jammy [12.3 MB] 2023-10-21T04:03:15.058 INFO:teuthology.orchestra.run.smithi172.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6744-g7f103c7c-1jammy [91.6 kB] 2023-10-21T04:03:15.609 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 184 MB in 6s (30.6 MB/s) 2023-10-21T04:03:15.748 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-21T04:03:15.844 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165984 files and directories currently installed.) 2023-10-21T04:03:15.849 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-21T04:03:15.930 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-21T04:03:16.258 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-21T04:03:16.284 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-21T04:03:16.325 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-21T04:03:16.670 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-21T04:03:16.696 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-21T04:03:16.736 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-21T04:03:17.023 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-21T04:03:17.049 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-21T04:03:17.081 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-21T04:03:17.509 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-21T04:03:17.536 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-21T04:03:17.584 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:03:17.996 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-21T04:03:18.022 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-21T04:03:18.063 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:03:18.366 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-21T04:03:18.392 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-21T04:03:18.424 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:03:18.777 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-21T04:03:18.803 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-21T04:03:18.844 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-21T04:03:19.163 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librados2. 2023-10-21T04:03:19.190 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../008-librados2_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:19.230 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librados2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:19.666 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librbd1. 2023-10-21T04:03:19.692 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../009-librbd1_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:19.733 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librbd1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:20.377 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs2. 2023-10-21T04:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:20.721 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rados. 2023-10-21T04:03:20.737 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:20.780 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rados (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:21.074 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-21T04:03:21.089 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:21.124 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-argparse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:21.326 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cephfs. 2023-10-21T04:03:21.353 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:21.393 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cephfs (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:21.713 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-21T04:03:21.739 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:21.779 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:22.091 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-21T04:03:22.117 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-21T04:03:22.157 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-21T04:03:22.452 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-prettytable. 2023-10-21T04:03:22.467 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-21T04:03:22.502 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-21T04:03:22.738 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rbd. 2023-10-21T04:03:22.753 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:22.797 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rbd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:23.141 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-21T04:03:23.167 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-21T04:03:23.207 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-21T04:03:23.552 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librgw2. 2023-10-21T04:03:23.578 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../019-librgw2_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:23.618 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librgw2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:24.213 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rgw. 2023-10-21T04:03:24.228 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:24.262 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:24.574 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-21T04:03:24.600 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-21T04:03:24.640 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-21T04:03:24.943 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libradosstriper1. 2023-10-21T04:03:24.969 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:25.018 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libradosstriper1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:25.489 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-common. 2023-10-21T04:03:25.515 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:25.555 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:26.811 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-base. 2023-10-21T04:03:26.837 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:26.886 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-base (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:27.623 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-21T04:03:27.650 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-21T04:03:27.690 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-21T04:03:28.001 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cheroot. 2023-10-21T04:03:28.027 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-21T04:03:28.068 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-21T04:03:28.370 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-21T04:03:28.397 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-21T04:03:28.446 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-21T04:03:28.715 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-21T04:03:28.742 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-21T04:03:28.798 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-21T04:03:29.109 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-21T04:03:29.136 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-21T04:03:29.176 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-21T04:03:29.437 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempora. 2023-10-21T04:03:29.453 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-21T04:03:29.496 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-21T04:03:29.799 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-portend. 2023-10-21T04:03:29.825 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-21T04:03:29.865 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-21T04:03:30.168 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-21T04:03:30.193 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-21T04:03:30.234 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-21T04:03:30.538 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-21T04:03:30.564 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-21T04:03:30.604 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-21T04:03:30.924 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-natsort. 2023-10-21T04:03:30.951 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-21T04:03:30.983 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-21T04:03:31.311 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-logutils. 2023-10-21T04:03:31.337 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-21T04:03:31.377 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-21T04:03:31.689 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-21T04:03:31.715 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-21T04:03:31.755 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-21T04:03:32.050 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-mako. 2023-10-21T04:03:32.076 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-21T04:03:32.116 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-21T04:03:32.420 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-21T04:03:32.446 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-21T04:03:32.478 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-21T04:03:32.773 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-21T04:03:32.799 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-21T04:03:32.839 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-21T04:03:33.142 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webob. 2023-10-21T04:03:33.168 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-21T04:03:33.208 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-21T04:03:33.536 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-waitress. 2023-10-21T04:03:33.563 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-21T04:03:33.606 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-21T04:03:33.906 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempita. 2023-10-21T04:03:33.932 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-21T04:03:33.964 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-21T04:03:34.259 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-paste. 2023-10-21T04:03:34.286 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-21T04:03:34.325 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-21T04:03:34.603 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-21T04:03:34.618 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-21T04:03:34.653 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-21T04:03:34.931 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-21T04:03:34.958 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-21T04:03:35.014 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-21T04:03:35.318 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webtest. 2023-10-21T04:03:35.344 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-21T04:03:35.384 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-21T04:03:35.929 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pecan. 2023-10-21T04:03:35.955 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-21T04:03:35.995 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-21T04:03:36.350 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-certifi. 2023-10-21T04:03:36.377 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-21T04:03:36.417 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-21T04:03:36.753 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-idna. 2023-10-21T04:03:36.779 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-21T04:03:36.819 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-21T04:03:37.081 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-urllib3. 2023-10-21T04:03:37.096 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-21T04:03:37.131 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-21T04:03:37.400 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-requests. 2023-10-21T04:03:37.416 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-21T04:03:37.450 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-21T04:03:37.762 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-21T04:03:37.788 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-21T04:03:37.828 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-21T04:03:38.125 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-21T04:03:38.140 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:38.183 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:38.546 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-21T04:03:38.573 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:38.613 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:38.916 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-openssl. 2023-10-21T04:03:38.942 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-21T04:03:38.982 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-21T04:03:39.294 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr. 2023-10-21T04:03:39.320 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:39.360 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:39.747 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mon. 2023-10-21T04:03:39.773 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:39.813 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mon (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:40.667 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-osd. 2023-10-21T04:03:40.694 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:40.733 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-osd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:42.545 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph. 2023-10-21T04:03:42.572 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../059-ceph_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:42.611 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:43.615 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-fuse. 2023-10-21T04:03:43.642 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:43.690 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:44.043 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mds. 2023-10-21T04:03:44.070 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:44.109 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mds (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:44.496 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package cephadm. 2023-10-21T04:03:44.523 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../062-cephadm_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:44.562 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:44.816 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-21T04:03:44.843 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-21T04:03:44.882 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-21T04:03:45.160 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-21T04:03:45.176 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-21T04:03:45.210 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-21T04:03:46.374 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-babel. 2023-10-21T04:03:46.401 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-21T04:03:46.440 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-21T04:03:46.776 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jinja2. 2023-10-21T04:03:46.803 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-21T04:03:46.843 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-21T04:03:47.205 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-21T04:03:47.232 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:47.271 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:47.608 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-21T04:03:47.635 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-21T04:03:47.674 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-21T04:03:47.977 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-21T04:03:48.006 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-21T04:03:48.044 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-21T04:03:48.297 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-routes. 2023-10-21T04:03:48.324 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-21T04:03:48.355 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-21T04:03:48.701 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-21T04:03:48.727 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:48.767 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:49.154 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-21T04:03:49.169 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-21T04:03:49.203 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-21T04:03:49.649 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-joblib. 2023-10-21T04:03:49.676 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-21T04:03:49.707 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-21T04:03:50.053 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-21T04:03:50.068 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-21T04:03:50.110 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-21T04:03:50.422 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn. 2023-10-21T04:03:50.449 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-21T04:03:50.488 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-21T04:03:51.122 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-21T04:03:51.138 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:51.180 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:52.142 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cachetools. 2023-10-21T04:03:52.169 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-21T04:03:52.208 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-21T04:03:52.511 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-21T04:03:52.538 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-21T04:03:52.578 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-21T04:03:52.898 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-21T04:03:52.925 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-21T04:03:52.956 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-21T04:03:53.217 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rsa. 2023-10-21T04:03:53.233 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-21T04:03:53.276 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-21T04:03:53.563 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-google-auth. 2023-10-21T04:03:53.586 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-21T04:03:53.621 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-21T04:03:53.957 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-21T04:03:53.984 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-21T04:03:54.024 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-21T04:03:54.387 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-websocket. 2023-10-21T04:03:54.411 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-21T04:03:54.445 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-21T04:03:54.748 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-21T04:03:54.776 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-21T04:03:54.815 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-21T04:03:55.454 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-21T04:03:55.482 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:55.520 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:55.848 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-21T04:03:55.875 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-21T04:03:55.906 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-21T04:03:56.235 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-21T04:03:56.262 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-21T04:03:56.301 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-21T04:03:56.729 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package jq. 2023-10-21T04:03:56.757 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-21T04:03:56.795 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-21T04:03:57.048 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package socat. 2023-10-21T04:03:57.064 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-21T04:03:57.106 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-21T04:03:57.402 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package xmlstarlet. 2023-10-21T04:03:57.429 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-21T04:03:57.468 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-21T04:03:57.823 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-test. 2023-10-21T04:03:57.850 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:57.889 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-test (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:00.160 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-volume. 2023-10-21T04:04:00.187 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:04:00.226 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-volume (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:00.513 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-21T04:04:00.529 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:04:00.571 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs-dev (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:00.900 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package nvme-cli. 2023-10-21T04:04:00.928 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-21T04:04:00.966 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-21T04:04:01.413 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-21T04:04:01.441 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-21T04:04:01.471 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-21T04:04:01.900 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-21T04:04:01.927 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-21T04:04:01.966 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-21T04:04:02.236 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastescript. 2023-10-21T04:04:02.263 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-21T04:04:02.302 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-21T04:04:02.622 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pluggy. 2023-10-21T04:04:02.650 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-21T04:04:02.689 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-21T04:04:02.958 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-psutil. 2023-10-21T04:04:02.978 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-21T04:04:03.008 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-21T04:04:03.320 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-py. 2023-10-21T04:04:03.347 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-21T04:04:03.386 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-21T04:04:03.706 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pygments. 2023-10-21T04:04:03.734 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-21T04:04:03.772 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-21T04:04:04.235 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-21T04:04:04.263 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-21T04:04:04.318 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-21T04:04:04.604 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-toml. 2023-10-21T04:04:04.632 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-21T04:04:04.670 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-21T04:04:04.948 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pytest. 2023-10-21T04:04:04.976 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-21T04:04:05.015 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-21T04:04:05.352 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplejson. 2023-10-21T04:04:05.379 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-21T04:04:05.418 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-21T04:04:05.707 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-21T04:04:05.735 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-21T04:04:05.771 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-21T04:04:06.409 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package radosgw. 2023-10-21T04:04:06.436 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../107-radosgw_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:04:06.475 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking radosgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:07.236 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package rbd-fuse. 2023-10-21T04:04:07.252 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:04:07.294 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking rbd-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:07.598 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-21T04:04:07.849 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-21T04:04:07.975 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-21T04:04:08.239 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-21T04:04:08.590 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-21T04:04:09.076 INFO:teuthology.orchestra.run.smithi172.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-21T04:04:09.177 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:09.829 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user cephadm....done 2023-10-21T04:04:09.976 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-21T04:04:10.553 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-21T04:04:10.788 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-21T04:04:10.897 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-21T04:04:11.157 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-21T04:04:11.417 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-21T04:04:11.542 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-21T04:04:11.966 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-21T04:04:12.512 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-21T04:04:12.747 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-21T04:04:13.014 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-21T04:04:13.266 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-21T04:04:13.525 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-21T04:04:13.776 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-argparse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:14.036 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-21T04:04:14.162 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-21T04:04:14.388 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-21T04:04:14.489 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-21T04:04:14.848 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-21T04:04:14.974 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-21T04:04:15.251 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-21T04:04:15.518 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-21T04:04:15.828 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-21T04:04:16.446 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-21T04:04:16.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-21T04:04:16.832 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-21T04:04:16.975 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-21T04:04:17.276 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-21T04:04:17.660 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-21T04:04:17.912 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-21T04:04:18.263 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-21T04:04:18.548 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-21T04:04:18.874 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-21T04:04:19.125 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-21T04:04:19.251 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:19.552 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-idna (3.3-1) ... 2023-10-21T04:04:19.853 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-21T04:04:20.959 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:04:21.118 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-21T04:04:21.378 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-21T04:04:21.479 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-21T04:04:21.597 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-21T04:04:21.898 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-21T04:04:22.157 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-21T04:04:22.475 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:04:22.601 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-21T04:04:22.727 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-21T04:04:23.004 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-21T04:04:23.246 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-21T04:04:23.506 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-21T04:04:23.759 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-21T04:04:24.010 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-21T04:04:24.261 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-21T04:04:24.362 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-21T04:04:24.613 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-21T04:04:24.739 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-21T04:04:25.040 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-21T04:04:25.333 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-21T04:04:25.601 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-21T04:04:26.003 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-21T04:04:26.237 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-21T04:04:26.338 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-21T04:04:26.723 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-21T04:04:26.982 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-21T04:04:27.234 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:04:27.360 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-21T04:04:27.677 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-21T04:04:27.953 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-21T04:04:28.213 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-21T04:04:28.339 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-21T04:04:28.617 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-21T04:04:28.891 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-21T04:04:29.167 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-21T04:04:29.569 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-21T04:04:29.845 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-21T04:04:30.180 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-21T04:04:30.306 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librados2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:30.433 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librgw2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:30.558 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:30.693 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:30.819 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libradosstriper1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:31.030 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-21T04:04:31.323 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librbd1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:31.449 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-modules-core (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:31.576 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:31.792 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-21T04:04:31.793 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-21T04:04:32.369 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs-dev (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:32.470 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rados (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:32.596 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-21T04:04:33.834 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rbd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:33.960 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up rbd-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:34.061 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:34.161 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cephfs (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:34.262 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:34.814 INFO:teuthology.orchestra.run.smithi172.stdout:Adding group ceph....done 2023-10-21T04:04:35.198 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user ceph....done 2023-10-21T04:04:35.375 INFO:teuthology.orchestra.run.smithi172.stdout:Setting system user ceph properties....done 2023-10-21T04:04:35.394 INFO:teuthology.orchestra.run.smithi172.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-21T04:04:35.532 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-21T04:04:35.861 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-21T04:04:36.438 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-test (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:36.565 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up radosgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:37.087 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-21T04:04:37.087 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-21T04:04:37.633 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-base (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:38.229 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-21T04:04:38.751 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mds (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:38.953 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-21T04:04:38.953 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-21T04:04:39.470 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:39.671 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-21T04:04:39.671 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-21T04:04:40.213 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-osd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:40.544 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-21T04:04:40.544 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-21T04:04:41.072 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:41.198 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:41.358 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mon (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:41.570 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-21T04:04:41.571 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-21T04:04:42.117 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:42.243 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:42.369 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-dashboard (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:42.520 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-volume (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:42.864 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-21T04:04:44.162 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-21T04:04:44.609 INFO:teuthology.orchestra.run.smithi172.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-21T04:04:44.612 DEBUG:teuthology.parallel:result is None 2023-10-21T04:04:44.612 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:04:44.840 DEBUG:teuthology.orchestra.run.smithi172:> dpkg-query -W -f '${Version}' ceph 2023-10-21T04:04:44.859 INFO:teuthology.orchestra.run.smithi172.stdout:18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:04:44.860 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:04:44.860 INFO:teuthology.task.install:The correct ceph version 18.0.0-6744-g7f103c7c-1jammy is installed. 2023-10-21T04:04:44.861 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-21T04:04:44.861 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:44.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-21T04:04:44.914 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-21T04:04:44.915 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:44.915 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2023-10-21T04:04:44.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-21T04:04:45.020 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-21T04:04:45.021 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:45.021 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-21T04:04:45.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-21T04:04:45.124 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-21T04:04:45.125 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:45.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2023-10-21T04:04:45.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-21T04:04:45.230 INFO:teuthology.run_tasks:Running task ceph... 2023-10-21T04:04:45.308 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-21T04:04:45.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2023-10-21T04:04:45.323 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-21T04:04:45.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-21T04:04:45.377 INFO:tasks.ceph:Creating extra log directories... 2023-10-21T04:04:45.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-21T04:04:45.437 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-21T04:04:45.438 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'enable experimental unrecoverable data corrupting features': '*', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug randomize hobject sort order': False, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-21T04:04:45.438 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433369', 'branch': 'ajarr-wip-extra-debug-62994-oct-19', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/ec features/layering msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7433369', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'ajarr-wip-extra-debug-62994-oct-19'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'enable experimental unrecoverable data corrupting features': '*', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug randomize hobject sort order': False, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'ajarr-wip-extra-debug-62994-oct-19', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1449, 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'ajarr-wip-extra-debug-62994-oct-19', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab', 'targets': {'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPC+zfLfb5HR102ZeiDb68RvA1E5FrxGqtkVP/oLWYY5xf6lz9rSaD7Wy5qIbUtrDsS7z0KZ/yJdQAhgMzPfMtY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd erasure-code-profile set teuthologyprofile crush-failure-domain=osd m=1 k=2', 'sudo ceph osd pool create datapool 4 4 erasure teuthologyprofile', 'sudo ceph osd pool set datapool allow_ec_overwrites true', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '202b180cb047e798fb131047314a862593f45403', 'timestamp': '2023-10-20_15:15:00', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-21T04:04:45.439 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-21T04:04:45.478 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2023-10-21T04:04:45.534 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:45.534 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2023-10-21T04:04:45.582 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-21T04:04:45.582 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2023-10-21T04:04:45.630 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-21T04:04:45.630 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T04:04:45.630 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 884 Links: 1 2023-10-21T04:04:45.631 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T04:04:45.631 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-21 03:59:40.163308270 +0000 2023-10-21T04:04:45.631 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-21 03:59:39.771311771 +0000 2023-10-21T04:04:45.631 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-21 03:59:39.771311771 +0000 2023-10-21T04:04:45.631 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-21T04:04:45.632 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-21T04:04:45.684 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-21T04:04:45.685 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-21T04:04:45.685 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000458496 s, 1.1 MB/s 2023-10-21T04:04:45.686 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-21T04:04:45.735 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2023-10-21T04:04:45.781 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-21T04:04:45.782 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T04:04:45.782 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 904 Links: 1 2023-10-21T04:04:45.782 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T04:04:45.782 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-21 03:59:40.415306018 +0000 2023-10-21T04:04:45.783 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-21 03:59:40.411306055 +0000 2023-10-21T04:04:45.783 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-21 03:59:40.411306055 +0000 2023-10-21T04:04:45.783 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-21T04:04:45.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-21T04:04:45.835 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-21T04:04:45.836 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-21T04:04:45.836 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000411184 s, 1.2 MB/s 2023-10-21T04:04:45.837 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-21T04:04:45.887 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2023-10-21T04:04:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-21T04:04:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T04:04:45.934 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 915 Links: 1 2023-10-21T04:04:45.934 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T04:04:45.935 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-21 03:59:41.415297087 +0000 2023-10-21T04:04:45.935 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-21 03:59:41.039300446 +0000 2023-10-21T04:04:45.935 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-21 03:59:41.039300446 +0000 2023-10-21T04:04:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-21T04:04:45.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-21T04:04:45.988 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-21T04:04:45.988 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-21T04:04:45.988 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000411831 s, 1.2 MB/s 2023-10-21T04:04:45.989 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-21T04:04:46.039 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2023-10-21T04:04:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-21T04:04:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T04:04:46.087 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 935 Links: 1 2023-10-21T04:04:46.087 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T04:04:46.087 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-21 03:59:42.095291013 +0000 2023-10-21T04:04:46.087 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-21 03:59:41.675294765 +0000 2023-10-21T04:04:46.087 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-21 03:59:41.675294765 +0000 2023-10-21T04:04:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-21T04:04:46.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-21T04:04:46.140 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-21T04:04:46.140 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-21T04:04:46.140 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000277561 s, 1.8 MB/s 2023-10-21T04:04:46.141 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-21T04:04:46.191 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T04:04:46.191 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-21T04:04:46.192 INFO:tasks.ceph:Generating config... 2023-10-21T04:04:46.193 INFO:tasks.ceph:[client] rbd default data pool = datapool 2023-10-21T04:04:46.193 INFO:tasks.ceph:[client] rbd default features = 1 2023-10-21T04:04:46.193 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-21T04:04:46.194 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-21T04:04:46.194 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-21T04:04:46.194 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-21T04:04:46.195 INFO:tasks.ceph:[mgr] debug monc = 20 2023-10-21T04:04:46.195 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-21T04:04:46.195 INFO:tasks.ceph:[mgr] debug objecter = 20 2023-10-21T04:04:46.195 INFO:tasks.ceph:[mgr] debug rados = 20 2023-10-21T04:04:46.196 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-21T04:04:46.196 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-21T04:04:46.196 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-21T04:04:46.196 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-21T04:04:46.196 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-21T04:04:46.197 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-21T04:04:46.197 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-21T04:04:46.197 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-21T04:04:46.197 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-21T04:04:46.198 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-21T04:04:46.198 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-21T04:04:46.198 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-21T04:04:46.198 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-21T04:04:46.199 INFO:tasks.ceph:[osd] enable experimental unrecoverable data corrupting features = * 2023-10-21T04:04:46.199 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-21T04:04:46.199 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-21T04:04:46.199 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-21T04:04:46.200 INFO:tasks.ceph:[osd] osd debug randomize hobject sort order = False 2023-10-21T04:04:46.200 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-21T04:04:46.200 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-10-21T04:04:46.200 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-21T04:04:46.201 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-21T04:04:46.201 INFO:tasks.ceph:Setting up mon.a... 2023-10-21T04:04:46.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-21T04:04:46.268 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.keyring 2023-10-21T04:04:46.280 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-21T04:04:46.354 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-21T04:04:46.408 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.172')] 2023-10-21T04:04:46.408 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.172', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'enable experimental unrecoverable data corrupting features': '*', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug randomize hobject sort order': False, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 20, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 1}, 'mon.a': {}} 2023-10-21T04:04:46.409 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:46.410 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-21T04:04:46.453 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.172 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:46.531 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:46.531 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: generated fsid e51e76a3-8801-445f-ab5b-9b66dbc6ff69 2023-10-21T04:04:46.531 INFO:teuthology.orchestra.run.smithi172.stdout:setting min_mon_release = pacific 2023-10-21T04:04:46.532 INFO:teuthology.orchestra.run.smithi172.stdout:epoch 0 2023-10-21T04:04:46.532 INFO:teuthology.orchestra.run.smithi172.stdout:fsid e51e76a3-8801-445f-ab5b-9b66dbc6ff69 2023-10-21T04:04:46.532 INFO:teuthology.orchestra.run.smithi172.stdout:last_changed 2023-10-21T04:04:46.530473+0000 2023-10-21T04:04:46.532 INFO:teuthology.orchestra.run.smithi172.stdout:created 2023-10-21T04:04:46.530473+0000 2023-10-21T04:04:46.532 INFO:teuthology.orchestra.run.smithi172.stdout:min_mon_release 16 (pacific) 2023-10-21T04:04:46.533 INFO:teuthology.orchestra.run.smithi172.stdout:election_strategy: 1 2023-10-21T04:04:46.533 INFO:teuthology.orchestra.run.smithi172.stdout:0: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.a 2023-10-21T04:04:46.533 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-21T04:04:46.534 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-21T04:04:46.582 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e51e76a3-8801-445f-ab5b-9b66dbc6ff69... 2023-10-21T04:04:46.583 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-21T04:04:46.655 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2023-10-21T04:04:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2023-10-21T04:04:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-21T04:04:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2023-10-21T04:04:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2023-10-21T04:04:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2023-10-21T04:04:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2023-10-21T04:04:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2023-10-21T04:04:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2023-10-21T04:04:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2023-10-21T04:04:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-21T04:04:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2023-10-21T04:04:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2023-10-21T04:04:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-21T04:04:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2023-10-21T04:04:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2023-10-21T04:04:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2023-10-21T04:04:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2023-10-21T04:04:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2023-10-21T04:04:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2023-10-21T04:04:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2023-10-21T04:04:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2023-10-21T04:04:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-21T04:04:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-21T04:04:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2023-10-21T04:04:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-21T04:04:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2023-10-21T04:04:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2023-10-21T04:04:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2023-10-21T04:04:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2023-10-21T04:04:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2023-10-21T04:04:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.172 2023-10-21T04:04:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: mon client directed command retry = 5 2023-10-21T04:04:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on pool no app = False 2023-10-21T04:04:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject socket failures = 5000 2023-10-21T04:04:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = e51e76a3-8801-445f-ab5b-9b66dbc6ff69 2023-10-21T04:04:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.665 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2023-10-21T04:04:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2023-10-21T04:04:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2023-10-21T04:04:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2023-10-21T04:04:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2023-10-21T04:04:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2023-10-21T04:04:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2023-10-21T04:04:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2023-10-21T04:04:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2023-10-21T04:04:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2023-10-21T04:04:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2023-10-21T04:04:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2023-10-21T04:04:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2023-10-21T04:04:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2023-10-21T04:04:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2023-10-21T04:04:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2023-10-21T04:04:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2023-10-21T04:04:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2023-10-21T04:04:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2023-10-21T04:04:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: bdev async discard = True 2023-10-21T04:04:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: bdev enable discard = True 2023-10-21T04:04:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2023-10-21T04:04:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2023-10-21T04:04:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 1/20 2023-10-21T04:04:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 1/20 2023-10-21T04:04:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-21T04:04:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2023-10-21T04:04:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 4/10 2023-10-21T04:04:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: enable experimental unrecoverable data corrupting features = * 2023-10-21T04:04:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2023-10-21T04:04:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2023-10-21T04:04:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2023-10-21T04:04:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug randomize hobject sort order = False 2023-10-21T04:04:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2023-10-21T04:04:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: osd memory target = 1610612736 2023-10-21T04:04:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2023-10-21T04:04:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: osd shutdown pgref assert = True 2023-10-21T04:04:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.676 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2023-10-21T04:04:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-21T04:04:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2023-10-21T04:04:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-10-21T04:04:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-10-21T04:04:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-10-21T04:04:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-10-21T04:04:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2023-10-21T04:04:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: debug monc = 20 2023-10-21T04:04:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: debug objecter = 20 2023-10-21T04:04:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: debug rados = 20 2023-10-21T04:04:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: debug rbd = 20 2023-10-21T04:04:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.678 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2023-10-21T04:04:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-21T04:04:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-10-21T04:04:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2023-10-21T04:04:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-10-21T04:04:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2023-10-21T04:04:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2023-10-21T04:04:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-10-21T04:04:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2023-10-21T04:04:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2023-10-21T04:04:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-10-21T04:04:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-21T04:04:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2023-10-21T04:04:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2023-10-21T04:04:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2023-10-21T04:04:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-21T04:04:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-21T04:04:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2023-10-21T04:04:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2023-10-21T04:04:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2023-10-21T04:04:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:04:46.683 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2023-10-21T04:04:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2023-10-21T04:04:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2023-10-21T04:04:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2023-10-21T04:04:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-21T04:04:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-21T04:04:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: rbd default data pool = datapool 2023-10-21T04:04:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: rbd default features = 1 2023-10-21T04:04:46.685 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2023-10-21T04:04:46.686 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-21T04:04:46.686 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-21T04:04:46.777 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-21T04:04:46.777 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:46.778 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-21T04:04:46.783 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:46.784 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-21T04:04:46.830 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2023-10-21T04:04:46.830 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:46.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-21T04:04:46.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-21T04:04:46.895 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:46.895 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:46.942 INFO:tasks.ceph:Setting up mon nodes... 2023-10-21T04:04:46.942 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-21T04:04:46.943 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-21T04:04:47.026 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-21T04:04:47.029 INFO:tasks.ceph:Setting up mds nodes... 2023-10-21T04:04:47.029 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-21T04:04:47.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-21T04:04:47.105 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-21T04:04:47.115 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-21T04:04:47.115 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-21T04:04:47.116 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-21T04:04:47.167 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T04:04:47.168 INFO:tasks.ceph:role: osd.0 2023-10-21T04:04:47.168 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-21T04:04:47.169 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-21T04:04:47.230 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-21T04:04:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-21T04:04:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-21T04:04:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-21T04:04:47.231 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-21T04:04:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-21T04:04:47.231 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-21T04:04:47.231 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-21T04:04:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-21T04:04:47.232 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-21T04:04:47.688 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-21T04:04:47.689 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-21T04:04:47.690 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-21T04:04:47.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-21T04:04:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:04:47.795 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2023-10-21T04:04:47.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-21T04:04:47.849 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T04:04:47.850 INFO:tasks.ceph:role: osd.1 2023-10-21T04:04:47.850 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-21T04:04:47.850 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-21T04:04:47.914 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-21T04:04:47.914 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-21T04:04:47.915 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-21T04:04:47.915 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-21T04:04:47.915 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-21T04:04:47.915 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-21T04:04:47.915 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-21T04:04:47.915 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-21T04:04:47.916 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-21T04:04:47.916 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-21T04:04:48.381 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-21T04:04:48.383 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-21T04:04:48.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-21T04:04:48.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-21T04:04:48.553 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2023-10-21T04:04:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:04:48.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-21T04:04:48.609 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T04:04:48.610 INFO:tasks.ceph:role: osd.2 2023-10-21T04:04:48.610 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-21T04:04:48.610 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-21T04:04:48.674 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-21T04:04:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-21T04:04:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-21T04:04:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-21T04:04:48.675 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-21T04:04:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-21T04:04:48.675 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-21T04:04:48.675 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-21T04:04:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-21T04:04:48.676 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-21T04:04:49.156 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-21T04:04:49.157 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-21T04:04:49.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-21T04:04:49.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-21T04:04:49.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:04:49.323 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2023-10-21T04:04:49.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:49.431 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:49.426+0000 7f4c48832640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:04:49.432 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:49.426+0000 7f4c48832640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:04:49.433 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:49.426+0000 7f4c48832640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:04:49.434 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:49.430+0000 7f4c48832640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-21T04:04:49.434 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:49.430+0000 7f4c48832640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-21T04:04:49.435 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:49.430+0000 7f4c48832640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-21T04:04:49.435 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:49.430+0000 7f4c48832640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-21T04:04:49.435 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:49.430+0000 7f4c48832640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-21T04:04:49.435 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:49.430+0000 7f4c48832640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-21T04:04:52.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-21T04:04:52.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:52.546 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:52.542+0000 7fb3f1974640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:04:52.546 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:52.542+0000 7fb3f1974640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:04:52.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:52.542+0000 7fb3f1974640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:04:52.548 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:52.542+0000 7fb3f1974640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-21T04:04:52.548 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:52.542+0000 7fb3f1974640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-21T04:04:52.549 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:52.542+0000 7fb3f1974640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-21T04:04:52.549 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:52.542+0000 7fb3f1974640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-21T04:04:52.549 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:52.542+0000 7fb3f1974640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-21T04:04:52.550 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:52.546+0000 7fb3f1974640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-21T04:04:55.569 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-21T04:04:55.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:55.711 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:55.706+0000 7faf2ad60640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:04:55.711 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:55.706+0000 7faf2ad60640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:04:55.712 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:55.706+0000 7faf2ad60640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:04:55.712 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:55.706+0000 7faf2ad60640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-21T04:04:55.713 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:55.710+0000 7faf2ad60640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-21T04:04:55.713 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:55.710+0000 7faf2ad60640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-21T04:04:55.714 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:55.710+0000 7faf2ad60640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-21T04:04:55.714 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:55.710+0000 7faf2ad60640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-21T04:04:55.714 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-21T04:04:55.710+0000 7faf2ad60640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-21T04:04:58.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-21T04:04:58.786 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-21T04:04:58.787 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:58.787 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-21T04:04:58.844 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:58.845 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-21T04:04:58.901 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:58.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-21T04:04:58.957 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:58.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-21T04:04:59.017 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:04:59.017 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-21T04:04:59.066 INFO:tasks.ceph:Adding keys to all mons... 2023-10-21T04:04:59.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-21T04:04:59.119 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2023-10-21T04:04:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDfTTNlpE6NARAAeiS7HEU3axVYeHNl16jtXA== 2023-10-21T04:04:59.120 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2023-10-21T04:04:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDhTTNlr0jYGRAAzCOmIxJ6EnGpflvn4/a3+Q== 2023-10-21T04:04:59.120 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2023-10-21T04:04:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDkTTNlKEWoIBAAaKwOHY6lBe5fx1p2fMnmwg== 2023-10-21T04:04:59.121 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2023-10-21T04:04:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDnTTNlRTJ5KhAAdb/L8Q/Za4VcoP/BVWXkbQ== 2023-10-21T04:04:59.121 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2023-10-21T04:04:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDfTTNlU0s8BhAAxY0yLDYqa62k2b3dcZdGtQ== 2023-10-21T04:04:59.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-21T04:04:59.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-21T04:04:59.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-21T04:04:59.334 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-21T04:04:59.417 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-21T04:04:59.505 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-21T04:04:59.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-21T04:04:59.561 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-21T04:04:59.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-21T04:04:59.667 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:59.712 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-21T04:04:59.713 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-21T04:04:59.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-21T04:04:59.754 INFO:tasks.ceph.mon.a:Started 2023-10-21T04:04:59.755 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-21T04:04:59.755 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-21T04:04:59.756 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-21T04:04:59.757 INFO:tasks.ceph.mgr.x:Started 2023-10-21T04:04:59.758 INFO:tasks.ceph:Setting crush tunables to default 2023-10-21T04:04:59.758 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd crush tunables default 2023-10-21T04:05:00.002 INFO:teuthology.orchestra.run.smithi172.stderr:adjusted tunables profile to default 2023-10-21T04:05:00.022 INFO:tasks.ceph:check_enable_crimson: False 2023-10-21T04:05:00.023 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-21T04:05:00.023 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:05:00.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-21T04:05:00.035 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:05:00.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-21T04:05:00.092 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:05:00.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-21T04:05:00.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new cc8c080d-06c7-4b59-abca-a7dff121a69a 0 2023-10-21T04:05:00.154 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:00.150+0000 7fa82d70c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-21T04:05:00.352 INFO:teuthology.orchestra.run.smithi172.stdout:0 2023-10-21T04:05:00.369 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 1616f68c-54bc-414c-b3c8-07d7c4e24807 1 2023-10-21T04:05:00.560 INFO:teuthology.orchestra.run.smithi172.stdout:1 2023-10-21T04:05:00.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new e6a20f19-f003-4d2c-8574-778059175260 2 2023-10-21T04:05:00.677 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:00.670+0000 7fa82d70c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-21T04:05:00.777 INFO:teuthology.orchestra.run.smithi172.stdout:2 2023-10-21T04:05:00.794 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-21T04:05:00.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-21T04:05:00.797 INFO:tasks.ceph.osd.0:Started 2023-10-21T04:05:00.797 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-21T04:05:00.798 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-21T04:05:00.800 INFO:tasks.ceph.osd.1:Started 2023-10-21T04:05:00.800 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-21T04:05:00.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-21T04:05:00.802 INFO:tasks.ceph.osd.2:Started 2023-10-21T04:05:00.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T04:05:00.854 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:05:00.850+0000 7f7ebeb2a640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:05:00.855 INFO:tasks.ceph.osd.2.smithi172.stderr:2023-10-21T04:05:00.850+0000 7f93c34b2640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:05:00.859 INFO:tasks.ceph.osd.2.smithi172.stderr:2023-10-21T04:05:00.854+0000 7f93c34b2640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:05:00.859 INFO:tasks.ceph.osd.2.smithi172.stderr:2023-10-21T04:05:00.854+0000 7f93c34b2640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:05:00.860 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:05:00.854+0000 7f7ebeb2a640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:05:00.860 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:05:00.854+0000 7f7ebeb2a640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:05:00.865 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:05:00.862+0000 7f70b27fe640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:05:00.871 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:05:00.866+0000 7f70b27fe640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:05:00.871 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:05:00.866+0000 7f70b27fe640 -1 WARNING: all dangerous and experimental features are enabled. 2023-10-21T04:05:00.880 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:00.874+0000 7fa82d70c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-21T04:05:00.991 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:05:00.991 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":5,"fsid":"e51e76a3-8801-445f-ab5b-9b66dbc6ff69","created":"2023-10-21T04:04:59.896377+0000","modified":"2023-10-21T04:05:00.775907+0000","last_up_change":"0.000000","last_in_change":"2023-10-21T04:05:00.775907+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cc8c080d-06c7-4b59-abca-a7dff121a69a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1616f68c-54bc-414c-b3c8-07d7c4e24807","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e6a20f19-f003-4d2c-8574-778059175260","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:05:01.007 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-21T04:05:01.008 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-21T04:05:01.310 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-21T04:05:01.410 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:05:01.406+0000 7f7ebeb2a640 -1 Falling back to public interface 2023-10-21T04:05:01.414 INFO:tasks.ceph.osd.2.smithi172.stderr:2023-10-21T04:05:01.410+0000 7f93c34b2640 -1 Falling back to public interface 2023-10-21T04:05:01.430 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:05:01.426+0000 7f70b27fe640 -1 Falling back to public interface 2023-10-21T04:05:01.447 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:01.442+0000 7fa82d70c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-21T04:05:01.491 INFO:teuthology.misc.health.smithi172.stdout: 2023-10-21T04:05:01.491 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":5,"fsid":"e51e76a3-8801-445f-ab5b-9b66dbc6ff69","created":"2023-10-21T04:04:59.896377+0000","modified":"2023-10-21T04:05:00.775907+0000","last_up_change":"0.000000","last_in_change":"2023-10-21T04:05:00.775907+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cc8c080d-06c7-4b59-abca-a7dff121a69a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1616f68c-54bc-414c-b3c8-07d7c4e24807","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e6a20f19-f003-4d2c-8574-778059175260","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:05:01.502 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:01.498+0000 7fa82d70c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-21T04:05:01.507 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-21T04:05:01.555 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:01.550+0000 7fa82d70c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-21T04:05:01.631 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:01.626+0000 7fa82d70c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-21T04:05:01.682 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:01.678+0000 7fa82d70c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-21T04:05:01.758 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:01.754+0000 7fa82d70c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-21T04:05:01.809 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:01.802+0000 7fa82d70c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-21T04:05:01.947 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:01.942+0000 7fa82d70c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-21T04:05:02.006 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:02.002+0000 7fa82d70c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-21T04:05:02.655 INFO:tasks.ceph.mgr.x.smithi172.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-21T04:05:02.656 INFO:tasks.ceph.mgr.x.smithi172.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-21T04:05:02.656 INFO:tasks.ceph.mgr.x.smithi172.stderr: from numpy import show_config as show_numpy_config 2023-10-21T04:05:02.726 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:02.722+0000 7fa82d70c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-21T04:05:02.886 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:02.882+0000 7fa82d70c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-21T04:05:03.040 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:03.034+0000 7fa82d70c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-21T04:05:03.103 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:03.098+0000 7fa82d70c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-21T04:05:03.224 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:03.218+0000 7fa82d70c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-21T04:05:03.836 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:03.830+0000 7fa82d70c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-21T04:05:03.910 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:03.906+0000 7fa82d70c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-21T04:05:04.087 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:04.082+0000 7fa82d70c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-21T04:05:04.161 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:04.154+0000 7fa82d70c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-21T04:05:04.460 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:04.454+0000 7fa82d70c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-21T04:05:04.519 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:04.514+0000 7fa82d70c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-21T04:05:05.075 INFO:tasks.ceph.osd.2.smithi172.stderr:2023-10-21T04:05:05.070+0000 7f93c34b2640 -1 osd.2 0 log_to_monitors true 2023-10-21T04:05:05.075 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:05:05.070+0000 7f70b27fe640 -1 osd.0 0 log_to_monitors true 2023-10-21T04:05:05.177 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:05:05.174+0000 7f7ebeb2a640 -1 osd.1 0 log_to_monitors true 2023-10-21T04:05:05.554 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:05:05.550+0000 7fa7d4e81640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-21T04:05:07.809 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-21T04:05:08.232 INFO:teuthology.misc.health.smithi172.stdout: 2023-10-21T04:05:08.232 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":7,"fsid":"e51e76a3-8801-445f-ab5b-9b66dbc6ff69","created":"2023-10-21T04:04:59.896377+0000","modified":"2023-10-21T04:05:06.845231+0000","last_up_change":"0.000000","last_in_change":"2023-10-21T04:05:00.775907+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cc8c080d-06c7-4b59-abca-a7dff121a69a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1616f68c-54bc-414c-b3c8-07d7c4e24807","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e6a20f19-f003-4d2c-8574-778059175260","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:05:08.258 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-21T04:05:08.768 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:05:08.762+0000 7f70adf83640 -1 osd.0 0 waiting for initial osdmap 2023-10-21T04:05:08.779 INFO:tasks.ceph.osd.2.smithi172.stderr:2023-10-21T04:05:08.774+0000 7f93bec37640 -1 osd.2 0 waiting for initial osdmap 2023-10-21T04:05:08.801 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:05:08.798+0000 7f70a65ca640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-21T04:05:08.816 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:05:08.810+0000 7f7eba2af640 -1 osd.1 0 waiting for initial osdmap 2023-10-21T04:05:08.833 INFO:tasks.ceph.osd.2.smithi172.stderr:2023-10-21T04:05:08.830+0000 7f93b727e640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-21T04:05:08.849 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:05:08.846+0000 7f7eb28f6640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-21T04:05:14.561 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-21T04:05:14.854 INFO:teuthology.misc.health.smithi172.stdout: 2023-10-21T04:05:14.855 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":11,"fsid":"e51e76a3-8801-445f-ab5b-9b66dbc6ff69","created":"2023-10-21T04:04:59.896377+0000","modified":"2023-10-21T04:05:12.891016+0000","last_up_change":"2023-10-21T04:05:09.780590+0000","last_in_change":"2023-10-21T04:05:00.775907+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T04:05:10.624097+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cc8c080d-06c7-4b59-abca-a7dff121a69a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6817","nonce":3641467676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6819","nonce":3641467676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6823","nonce":3641467676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6821","nonce":3641467676}]},"public_addr":"172.21.15.172:6817/3641467676","cluster_addr":"172.21.15.172:6819/3641467676","heartbeat_back_addr":"172.21.15.172:6823/3641467676","heartbeat_front_addr":"172.21.15.172:6821/3641467676","state":["exists","up"]},{"osd":1,"uuid":"1616f68c-54bc-414c-b3c8-07d7c4e24807","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6801","nonce":1983746680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6803","nonce":1983746680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6807","nonce":1983746680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6805","nonce":1983746680}]},"public_addr":"172.21.15.172:6801/1983746680","cluster_addr":"172.21.15.172:6803/1983746680","heartbeat_back_addr":"172.21.15.172:6807/1983746680","heartbeat_front_addr":"172.21.15.172:6805/1983746680","state":["exists","up"]},{"osd":2,"uuid":"e6a20f19-f003-4d2c-8574-778059175260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6809","nonce":920067689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6811","nonce":920067689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6815","nonce":920067689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6813","nonce":920067689}]},"public_addr":"172.21.15.172:6809/920067689","cluster_addr":"172.21.15.172:6811/920067689","heartbeat_back_addr":"172.21.15.172:6815/920067689","heartbeat_front_addr":"172.21.15.172:6813/920067689","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:05:06.035423+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:05:06.181353+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:05:06.119226+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:05:14.872 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-21T04:05:14.872 INFO:tasks.ceph:Creating RBD pool 2023-10-21T04:05:14.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-21T04:05:15.908 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'rbd' created 2023-10-21T04:05:15.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-21T04:05:16.917 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-21T04:05:16.950 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-21T04:05:16.950 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-21T04:05:16.951 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-21T04:05:16.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-21T04:05:16.951 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-21T04:05:17.276 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:05:17.292 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":1207428622},{"type":"v1","addr":"172.21.15.172:6825","nonce":1207428622}]},"active_addr":"172.21.15.172:6825/1207428622","active_change":"2023-10-21T04:05:04.538091+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2422092663}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2810223425}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2863900873}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":3418793587}]}]} 2023-10-21T04:05:17.295 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-21T04:05:17.295 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-21T04:05:17.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T04:05:17.568 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:05:17.569 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":13,"fsid":"e51e76a3-8801-445f-ab5b-9b66dbc6ff69","created":"2023-10-21T04:04:59.896377+0000","modified":"2023-10-21T04:05:16.908287+0000","last_up_change":"2023-10-21T04:05:09.780590+0000","last_in_change":"2023-10-21T04:05:00.775907+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T04:05:10.624097+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-21T04:05:15.189413+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cc8c080d-06c7-4b59-abca-a7dff121a69a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6817","nonce":3641467676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6819","nonce":3641467676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6823","nonce":3641467676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6821","nonce":3641467676}]},"public_addr":"172.21.15.172:6817/3641467676","cluster_addr":"172.21.15.172:6819/3641467676","heartbeat_back_addr":"172.21.15.172:6823/3641467676","heartbeat_front_addr":"172.21.15.172:6821/3641467676","state":["exists","up"]},{"osd":1,"uuid":"1616f68c-54bc-414c-b3c8-07d7c4e24807","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6801","nonce":1983746680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6803","nonce":1983746680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6807","nonce":1983746680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6805","nonce":1983746680}]},"public_addr":"172.21.15.172:6801/1983746680","cluster_addr":"172.21.15.172:6803/1983746680","heartbeat_back_addr":"172.21.15.172:6807/1983746680","heartbeat_front_addr":"172.21.15.172:6805/1983746680","state":["exists","up"]},{"osd":2,"uuid":"e6a20f19-f003-4d2c-8574-778059175260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6809","nonce":920067689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6811","nonce":920067689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6815","nonce":920067689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6813","nonce":920067689}]},"public_addr":"172.21.15.172:6809/920067689","cluster_addr":"172.21.15.172:6811/920067689","heartbeat_back_addr":"172.21.15.172:6815/920067689","heartbeat_front_addr":"172.21.15.172:6813/920067689","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:05:06.035423+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:05:06.181353+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:05:06.119226+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:05:17.584 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-21T04:05:17.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T04:05:17.871 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:05:17.872 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":13,"fsid":"e51e76a3-8801-445f-ab5b-9b66dbc6ff69","created":"2023-10-21T04:04:59.896377+0000","modified":"2023-10-21T04:05:16.908287+0000","last_up_change":"2023-10-21T04:05:09.780590+0000","last_in_change":"2023-10-21T04:05:00.775907+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T04:05:10.624097+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-21T04:05:15.189413+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cc8c080d-06c7-4b59-abca-a7dff121a69a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6817","nonce":3641467676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6819","nonce":3641467676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6823","nonce":3641467676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6821","nonce":3641467676}]},"public_addr":"172.21.15.172:6817/3641467676","cluster_addr":"172.21.15.172:6819/3641467676","heartbeat_back_addr":"172.21.15.172:6823/3641467676","heartbeat_front_addr":"172.21.15.172:6821/3641467676","state":["exists","up"]},{"osd":1,"uuid":"1616f68c-54bc-414c-b3c8-07d7c4e24807","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6801","nonce":1983746680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6803","nonce":1983746680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6807","nonce":1983746680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6805","nonce":1983746680}]},"public_addr":"172.21.15.172:6801/1983746680","cluster_addr":"172.21.15.172:6803/1983746680","heartbeat_back_addr":"172.21.15.172:6807/1983746680","heartbeat_front_addr":"172.21.15.172:6805/1983746680","state":["exists","up"]},{"osd":2,"uuid":"e6a20f19-f003-4d2c-8574-778059175260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6809","nonce":920067689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6811","nonce":920067689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6815","nonce":920067689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6813","nonce":920067689}]},"public_addr":"172.21.15.172:6809/920067689","cluster_addr":"172.21.15.172:6811/920067689","heartbeat_back_addr":"172.21.15.172:6815/920067689","heartbeat_front_addr":"172.21.15.172:6813/920067689","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:05:06.035423+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:05:06.181353+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:05:06.119226+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:05:17.890 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-21T04:05:17.891 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-21T04:05:17.891 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-21T04:05:18.041 INFO:teuthology.orchestra.run.smithi172.stdout:34359738374 2023-10-21T04:05:18.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-21T04:05:18.056 INFO:teuthology.orchestra.run.smithi172.stdout:34359738373 2023-10-21T04:05:18.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-21T04:05:18.058 INFO:teuthology.orchestra.run.smithi172.stdout:34359738373 2023-10-21T04:05:18.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-21T04:05:18.312 INFO:teuthology.orchestra.run.smithi172.stdout:34359738373 2023-10-21T04:05:18.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.0 2023-10-21T04:05:18.372 INFO:teuthology.orchestra.run.smithi172.stdout:34359738372 2023-10-21T04:05:18.388 INFO:teuthology.orchestra.run.smithi172.stdout:34359738372 2023-10-21T04:05:18.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-10-21T04:05:18.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-10-21T04:05:19.336 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-21T04:05:19.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-21T04:05:19.405 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-21T04:05:19.643 INFO:teuthology.orchestra.run.smithi172.stdout:34359738374 2023-10-21T04:05:19.660 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2023-10-21T04:05:19.660 DEBUG:teuthology.parallel:result is None 2023-10-21T04:05:19.681 INFO:teuthology.orchestra.run.smithi172.stdout:34359738373 2023-10-21T04:05:19.697 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-10-21T04:05:19.697 DEBUG:teuthology.parallel:result is None 2023-10-21T04:05:19.726 INFO:teuthology.orchestra.run.smithi172.stdout:34359738373 2023-10-21T04:05:19.742 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-10-21T04:05:19.742 DEBUG:teuthology.parallel:result is None 2023-10-21T04:05:19.742 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-21T04:05:19.743 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T04:05:20.053 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:05:20.053 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T04:05:20.070 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-21T04:05:18.546678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82564,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032956,"statfs":{"total":289910292480,"available":289825746944,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.636111"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.935446+0000","last_change":"2023-10-21T04:05:16.931966+0000","last_active":"2023-10-21T04:05:16.935446+0000","last_peered":"2023-10-21T04:05:16.935446+0000","last_clean":"2023-10-21T04:05:16.935446+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T04:05:16.935446+0000","last_undegraded":"2023-10-21T04:05:16.935446+0000","last_fullsized":"2023-10-21T04:05:16.935446+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:55:04.306272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.938316+0000","last_change":"2023-10-21T04:05:16.933396+0000","last_active":"2023-10-21T04:05:16.938316+0000","last_peered":"2023-10-21T04:05:16.938316+0000","last_clean":"2023-10-21T04:05:16.938316+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T04:05:16.938316+0000","last_undegraded":"2023-10-21T04:05:16.938316+0000","last_fullsized":"2023-10-21T04:05:16.938316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:14:23.399430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.935435+0000","last_change":"2023-10-21T04:05:16.931967+0000","last_active":"2023-10-21T04:05:16.935435+0000","last_peered":"2023-10-21T04:05:16.935435+0000","last_clean":"2023-10-21T04:05:16.935435+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T04:05:16.935435+0000","last_undegraded":"2023-10-21T04:05:16.935435+0000","last_fullsized":"2023-10-21T04:05:16.935435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:16:12.638382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.938302+0000","last_change":"2023-10-21T04:05:16.933170+0000","last_active":"2023-10-21T04:05:16.938302+0000","last_peered":"2023-10-21T04:05:16.938302+0000","last_clean":"2023-10-21T04:05:16.938302+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T04:05:16.938302+0000","last_undegraded":"2023-10-21T04:05:16.938302+0000","last_fullsized":"2023-10-21T04:05:16.938302+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:02:08.936692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:05:17.929433+0000","last_change":"2023-10-21T04:05:16.934828+0000","last_active":"2023-10-21T04:05:17.929433+0000","last_peered":"2023-10-21T04:05:17.929433+0000","last_clean":"2023-10-21T04:05:17.929433+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T04:05:17.929433+0000","last_undegraded":"2023-10-21T04:05:17.929433+0000","last_fullsized":"2023-10-21T04:05:17.929433+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:53:30.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.936542+0000","last_change":"2023-10-21T04:05:16.933866+0000","last_active":"2023-10-21T04:05:16.936542+0000","last_peered":"2023-10-21T04:05:16.936542+0000","last_clean":"2023-10-21T04:05:16.936542+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T04:05:16.936542+0000","last_undegraded":"2023-10-21T04:05:16.936542+0000","last_fullsized":"2023-10-21T04:05:16.936542+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:02:32.668840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.936503+0000","last_change":"2023-10-21T04:05:16.932868+0000","last_active":"2023-10-21T04:05:16.936503+0000","last_peered":"2023-10-21T04:05:16.936503+0000","last_clean":"2023-10-21T04:05:16.936503+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T04:05:16.936503+0000","last_undegraded":"2023-10-21T04:05:16.936503+0000","last_fullsized":"2023-10-21T04:05:16.936503+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:02:14.161620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.935465+0000","last_change":"2023-10-21T04:05:16.931948+0000","last_active":"2023-10-21T04:05:16.935465+0000","last_peered":"2023-10-21T04:05:16.935465+0000","last_clean":"2023-10-21T04:05:16.935465+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T04:05:16.935465+0000","last_undegraded":"2023-10-21T04:05:16.935465+0000","last_fullsized":"2023-10-21T04:05:16.935465+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:17:21.302544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-21T04:05:15.920862+0000","last_change":"2023-10-21T04:05:12.086073+0000","last_active":"2023-10-21T04:05:15.920862+0000","last_peered":"2023-10-21T04:05:15.920862+0000","last_clean":"2023-10-21T04:05:15.920862+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T04:05:15.920862+0000","last_undegraded":"2023-10-21T04:05:15.920862+0000","last_fullsized":"2023-10-21T04:05:15.920862+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:10.868962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:10.868962+0000","last_clean_scrub_stamp":"2023-10-21T04:05:10.868962+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:25:27.169589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T04:05:20.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T04:05:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:05:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T04:05:20.350 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-21T04:05:18.546678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82564,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032956,"statfs":{"total":289910292480,"available":289825746944,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.636111"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.935446+0000","last_change":"2023-10-21T04:05:16.931966+0000","last_active":"2023-10-21T04:05:16.935446+0000","last_peered":"2023-10-21T04:05:16.935446+0000","last_clean":"2023-10-21T04:05:16.935446+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T04:05:16.935446+0000","last_undegraded":"2023-10-21T04:05:16.935446+0000","last_fullsized":"2023-10-21T04:05:16.935446+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:55:04.306272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.938316+0000","last_change":"2023-10-21T04:05:16.933396+0000","last_active":"2023-10-21T04:05:16.938316+0000","last_peered":"2023-10-21T04:05:16.938316+0000","last_clean":"2023-10-21T04:05:16.938316+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T04:05:16.938316+0000","last_undegraded":"2023-10-21T04:05:16.938316+0000","last_fullsized":"2023-10-21T04:05:16.938316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:14:23.399430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.935435+0000","last_change":"2023-10-21T04:05:16.931967+0000","last_active":"2023-10-21T04:05:16.935435+0000","last_peered":"2023-10-21T04:05:16.935435+0000","last_clean":"2023-10-21T04:05:16.935435+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T04:05:16.935435+0000","last_undegraded":"2023-10-21T04:05:16.935435+0000","last_fullsized":"2023-10-21T04:05:16.935435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:16:12.638382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.938302+0000","last_change":"2023-10-21T04:05:16.933170+0000","last_active":"2023-10-21T04:05:16.938302+0000","last_peered":"2023-10-21T04:05:16.938302+0000","last_clean":"2023-10-21T04:05:16.938302+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T04:05:16.938302+0000","last_undegraded":"2023-10-21T04:05:16.938302+0000","last_fullsized":"2023-10-21T04:05:16.938302+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:02:08.936692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:05:17.929433+0000","last_change":"2023-10-21T04:05:16.934828+0000","last_active":"2023-10-21T04:05:17.929433+0000","last_peered":"2023-10-21T04:05:17.929433+0000","last_clean":"2023-10-21T04:05:17.929433+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T04:05:17.929433+0000","last_undegraded":"2023-10-21T04:05:17.929433+0000","last_fullsized":"2023-10-21T04:05:17.929433+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:53:30.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.936542+0000","last_change":"2023-10-21T04:05:16.933866+0000","last_active":"2023-10-21T04:05:16.936542+0000","last_peered":"2023-10-21T04:05:16.936542+0000","last_clean":"2023-10-21T04:05:16.936542+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T04:05:16.936542+0000","last_undegraded":"2023-10-21T04:05:16.936542+0000","last_fullsized":"2023-10-21T04:05:16.936542+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:02:32.668840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.936503+0000","last_change":"2023-10-21T04:05:16.932868+0000","last_active":"2023-10-21T04:05:16.936503+0000","last_peered":"2023-10-21T04:05:16.936503+0000","last_clean":"2023-10-21T04:05:16.936503+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T04:05:16.936503+0000","last_undegraded":"2023-10-21T04:05:16.936503+0000","last_fullsized":"2023-10-21T04:05:16.936503+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:02:14.161620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:16.935465+0000","last_change":"2023-10-21T04:05:16.931948+0000","last_active":"2023-10-21T04:05:16.935465+0000","last_peered":"2023-10-21T04:05:16.935465+0000","last_clean":"2023-10-21T04:05:16.935465+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T04:05:16.935465+0000","last_undegraded":"2023-10-21T04:05:16.935465+0000","last_fullsized":"2023-10-21T04:05:16.935465+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:17:21.302544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-21T04:05:15.920862+0000","last_change":"2023-10-21T04:05:12.086073+0000","last_active":"2023-10-21T04:05:15.920862+0000","last_peered":"2023-10-21T04:05:15.920862+0000","last_clean":"2023-10-21T04:05:15.920862+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T04:05:15.920862+0000","last_undegraded":"2023-10-21T04:05:15.920862+0000","last_fullsized":"2023-10-21T04:05:15.920862+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:10.868962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:10.868962+0000","last_clean_scrub_stamp":"2023-10-21T04:05:10.868962+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:25:27.169589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T04:05:20.352 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-21T04:05:20.352 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-21T04:05:20.352 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-21T04:05:20.353 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-21T04:05:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T04:05:20.642 INFO:teuthology.orchestra.run.smithi172.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-21T04:05:20.658 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-21T04:05:20.658 INFO:teuthology.run_tasks:Running task exec... 2023-10-21T04:05:20.668 INFO:teuthology.task.exec:Executing custom commands... 2023-10-21T04:05:20.668 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi172.front.sepia.ceph.com 2023-10-21T04:05:20.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd erasure-code-profile set teuthologyprofile crush-failure-domain=osd m=1 k=2' 2023-10-21T04:05:21.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4 4 erasure teuthologyprofile' 2023-10-21T04:05:23.957 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'datapool' created 2023-10-21T04:05:23.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set datapool allow_ec_overwrites true' 2023-10-21T04:05:24.970 INFO:teuthology.orchestra.run.smithi172.stderr:set pool 3 allow_ec_overwrites to true 2023-10-21T04:05:25.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2023-10-21T04:05:28.035 INFO:teuthology.run_tasks:Running task install... 2023-10-21T04:05:28.045 DEBUG:teuthology.task.install:project ceph 2023-10-21T04:05:28.045 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}} 2023-10-21T04:05:28.045 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'} 2023-10-21T04:05:28.046 INFO:teuthology.task.install:Using flavor: default 2023-10-21T04:05:28.052 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-21T04:05:28.052 INFO:teuthology.task.install:extra packages: [] 2023-10-21T04:05:28.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-key list | grep Ceph 2023-10-21T04:05:28.115 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-21T04:05:28.144 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-21T04:05:28.144 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph.com (release key) 2023-10-21T04:05:28.145 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-21T04:05:28.145 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2023-10-21T04:05:28.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:05:28.293 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default/ 2023-10-21T04:05:28.293 INFO:teuthology.task.install.deb:Package version is 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:05:28.465 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:05:28.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-21T04:05:28.480 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2023-10-21T04:05:28.767 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy InRelease 2023-10-21T04:05:28.780 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-21T04:05:28.804 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release 2023-10-21T04:05:28.841 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-21T04:05:28.852 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-21T04:05:28.936 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-21T04:05:29.149 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-21T04:05:30.218 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-10-21T04:05:30.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6744-g7f103c7c-1jammy cephadm=18.0.0-6744-g7f103c7c-1jammy ceph-mds=18.0.0-6744-g7f103c7c-1jammy ceph-mgr=18.0.0-6744-g7f103c7c-1jammy ceph-common=18.0.0-6744-g7f103c7c-1jammy ceph-fuse=18.0.0-6744-g7f103c7c-1jammy ceph-test=18.0.0-6744-g7f103c7c-1jammy ceph-volume=18.0.0-6744-g7f103c7c-1jammy radosgw=18.0.0-6744-g7f103c7c-1jammy python3-rados=18.0.0-6744-g7f103c7c-1jammy python3-rgw=18.0.0-6744-g7f103c7c-1jammy python3-cephfs=18.0.0-6744-g7f103c7c-1jammy python3-rbd=18.0.0-6744-g7f103c7c-1jammy libcephfs2=18.0.0-6744-g7f103c7c-1jammy libcephfs-dev=18.0.0-6744-g7f103c7c-1jammy librados2=18.0.0-6744-g7f103c7c-1jammy librbd1=18.0.0-6744-g7f103c7c-1jammy rbd-fuse=18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:05:30.294 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-10-21T04:05:30.478 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2023-10-21T04:05:30.479 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2023-10-21T04:05:30.639 INFO:teuthology.orchestra.run.smithi172.stdout:ceph is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.640 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-common is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.640 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-fuse is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.640 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-mds is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.640 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-mgr is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.640 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-test is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.641 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-volume is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.641 INFO:teuthology.orchestra.run.smithi172.stdout:cephadm is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.641 INFO:teuthology.orchestra.run.smithi172.stdout:libcephfs-dev is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.641 INFO:teuthology.orchestra.run.smithi172.stdout:libcephfs2 is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.641 INFO:teuthology.orchestra.run.smithi172.stdout:librados2 is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.642 INFO:teuthology.orchestra.run.smithi172.stdout:librbd1 is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.642 INFO:teuthology.orchestra.run.smithi172.stdout:python3-cephfs is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.642 INFO:teuthology.orchestra.run.smithi172.stdout:python3-rados is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.642 INFO:teuthology.orchestra.run.smithi172.stdout:python3-rbd is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.642 INFO:teuthology.orchestra.run.smithi172.stdout:python3-rgw is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.643 INFO:teuthology.orchestra.run.smithi172.stdout:radosgw is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.643 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-fuse is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:30.643 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T04:05:30.643 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-21T04:05:30.643 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-21T04:05:30.644 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-21T04:05:30.644 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-21T04:05:30.644 INFO:teuthology.orchestra.run.smithi172.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-21T04:05:30.644 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T04:05:30.676 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 0 newly installed, 0 to remove and 257 not upgraded. 2023-10-21T04:05:30.677 INFO:teuthology.orchestra.run.smithi172.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-21T04:05:30.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2023-10-21T04:05:30.737 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-10-21T04:05:30.925 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2023-10-21T04:05:30.926 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2023-10-21T04:05:31.085 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T04:05:31.086 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-10-21T04:05:31.086 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-10-21T04:05:31.086 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-10-21T04:05:31.086 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-10-21T04:05:31.087 INFO:teuthology.orchestra.run.smithi172.stdout: python2.7 python2.7-dev python2.7-minimal 2023-10-21T04:05:31.087 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T04:05:31.087 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2023-10-21T04:05:31.088 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemblk1 2023-10-21T04:05:31.088 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2023-10-21T04:05:31.089 INFO:teuthology.orchestra.run.smithi172.stdout: gnuplot gfio python-scipy 2023-10-21T04:05:31.113 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2023-10-21T04:05:31.114 INFO:teuthology.orchestra.run.smithi172.stdout: fio libpmemblk1 2023-10-21T04:05:31.293 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 2 newly installed, 0 to remove and 257 not upgraded. 2023-10-21T04:05:31.294 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 4,112 kB of archives. 2023-10-21T04:05:31.294 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 7,813 kB of additional disk space will be used. 2023-10-21T04:05:31.294 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-10-21T04:05:31.577 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-10-21T04:05:32.448 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 4,112 kB in 1s (4,272 kB/s) 2023-10-21T04:05:32.607 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-10-21T04:05:32.710 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172810 files and directories currently installed.) 2023-10-21T04:05:32.715 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-10-21T04:05:32.796 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-21T04:05:33.291 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package fio. 2023-10-21T04:05:33.319 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2023-10-21T04:05:33.358 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking fio (3.28-1) ... 2023-10-21T04:05:33.868 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-21T04:05:33.997 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up fio (3.28-1) ... 2023-10-21T04:05:34.491 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-21T04:05:35.517 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-21T04:05:35.996 INFO:teuthology.orchestra.run.smithi172.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-21T04:05:35.999 DEBUG:teuthology.parallel:result is None 2023-10-21T04:05:35.999 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:05:36.144 DEBUG:teuthology.orchestra.run.smithi172:> dpkg-query -W -f '${Version}' ceph 2023-10-21T04:05:36.163 INFO:teuthology.orchestra.run.smithi172.stdout:18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:05:36.164 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:05:36.164 INFO:teuthology.task.install:The correct ceph version 18.0.0-6744-g7f103c7c-1jammy is installed. 2023-10-21T04:05:36.165 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-21T04:05:36.165 INFO:teuthology.run_tasks:Running task workunit... 2023-10-21T04:05:36.178 INFO:tasks.workunit:Pulling workunits from ref 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:05:36.179 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-21T04:05:36.179 DEBUG:teuthology.orchestra.run.smithi172:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-21T04:05:36.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:05:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-21T04:05:36.211 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-21T04:05:36.257 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-21T04:05:36.258 DEBUG:teuthology.orchestra.run.smithi172:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-21T04:05:36.305 INFO:tasks.workunit:timeout=3h 2023-10-21T04:05:36.306 INFO:tasks.workunit:cleanup=True 2023-10-21T04:05:36.306 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:05:36.356 INFO:tasks.workunit.client.0.smithi172.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-21T04:06:36.517 INFO:tasks.workunit.client.0.smithi172.stderr:Note: switching to '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'. 2023-10-21T04:06:36.518 INFO:tasks.workunit.client.0.smithi172.stderr: 2023-10-21T04:06:36.518 INFO:tasks.workunit.client.0.smithi172.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-21T04:06:36.518 INFO:tasks.workunit.client.0.smithi172.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-21T04:06:36.518 INFO:tasks.workunit.client.0.smithi172.stderr:state without impacting any branches by switching back to a branch. 2023-10-21T04:06:36.519 INFO:tasks.workunit.client.0.smithi172.stderr: 2023-10-21T04:06:36.519 INFO:tasks.workunit.client.0.smithi172.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-21T04:06:36.519 INFO:tasks.workunit.client.0.smithi172.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-21T04:06:36.519 INFO:tasks.workunit.client.0.smithi172.stderr: 2023-10-21T04:06:36.519 INFO:tasks.workunit.client.0.smithi172.stderr: git switch -c 2023-10-21T04:06:36.520 INFO:tasks.workunit.client.0.smithi172.stderr: 2023-10-21T04:06:36.520 INFO:tasks.workunit.client.0.smithi172.stderr:Or undo this operation with: 2023-10-21T04:06:36.520 INFO:tasks.workunit.client.0.smithi172.stderr: 2023-10-21T04:06:36.520 INFO:tasks.workunit.client.0.smithi172.stderr: git switch - 2023-10-21T04:06:36.520 INFO:tasks.workunit.client.0.smithi172.stderr: 2023-10-21T04:06:36.521 INFO:tasks.workunit.client.0.smithi172.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-21T04:06:36.521 INFO:tasks.workunit.client.0.smithi172.stderr: 2023-10-21T04:06:36.521 INFO:tasks.workunit.client.0.smithi172.stderr:HEAD is now at 7f103c7cad2 qa/suites/rbd/cli/workloads/rbd_support_module_recovery: more debug 2023-10-21T04:06:36.522 DEBUG:teuthology.orchestra.run.smithi172:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-21T04:06:36.569 INFO:tasks.workunit.client.0.smithi172.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-21T04:06:36.572 INFO:tasks.workunit.client.0.smithi172.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-21T04:06:36.573 INFO:tasks.workunit.client.0.smithi172.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-21T04:06:37.547 INFO:tasks.workunit.client.0.smithi172.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-21T04:06:37.632 INFO:tasks.workunit.client.0.smithi172.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-21T04:06:37.683 INFO:tasks.workunit.client.0.smithi172.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-21T04:06:37.684 INFO:tasks.workunit.client.0.smithi172.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-21T04:06:37.685 INFO:tasks.workunit.client.0.smithi172.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-21T04:06:37.733 INFO:tasks.workunit.client.0.smithi172.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-21T04:06:37.736 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-21T04:06:37.737 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-21T04:06:37.784 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-21T04:06:37.786 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-21T04:06:37.786 DEBUG:teuthology.orchestra.run.smithi172:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-21T04:06:37.836 INFO:tasks.workunit.client.0.smithi172.stderr:+ POOL=rbd 2023-10-21T04:06:37.836 INFO:tasks.workunit.client.0.smithi172.stderr:+ IMAGE_PREFIX=image 2023-10-21T04:06:37.837 INFO:tasks.workunit.client.0.smithi172.stderr:+ NUM_IMAGES=20 2023-10-21T04:06:37.837 INFO:tasks.workunit.client.0.smithi172.stderr:+ RUN_TIME=3600 2023-10-21T04:06:37.837 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror pool enable rbd image 2023-10-21T04:06:37.881 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-21T04:06:37.918 INFO:tasks.workunit.client.0.smithi172.stdout:493f319a-f69e-44ab-a29c-3f91afed1301 2023-10-21T04:06:37.922 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i = 1 )) 2023-10-21T04:06:37.922 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:37.922 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image1 2023-10-21T04:06:37.970 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-21T04:06:38.965 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:38.978 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-21T04:06:39.040 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:39.040 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:39.041 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image2 2023-10-21T04:06:39.088 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-21T04:06:39.962 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:39.978 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-21T04:06:40.041 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:40.042 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:40.042 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image3 2023-10-21T04:06:40.089 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-21T04:06:40.965 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:40.981 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-21T04:06:41.043 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:41.043 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:41.043 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image4 2023-10-21T04:06:41.093 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-21T04:06:41.980 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:41.994 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-21T04:06:42.054 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:42.054 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:42.055 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image5 2023-10-21T04:06:42.103 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-21T04:06:42.987 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:43.003 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-21T04:06:43.064 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:43.064 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:43.065 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image6 2023-10-21T04:06:43.113 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-21T04:06:44.005 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:44.015 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-21T04:06:44.076 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:44.076 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:44.077 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image7 2023-10-21T04:06:44.126 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-21T04:06:45.010 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:45.022 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-21T04:06:45.082 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:45.082 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:45.082 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image8 2023-10-21T04:06:45.133 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-21T04:06:46.018 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:46.032 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-21T04:06:46.094 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:46.094 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:46.094 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image9 2023-10-21T04:06:46.145 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-21T04:06:47.027 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:47.039 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-21T04:06:47.102 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:47.102 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:47.102 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image10 2023-10-21T04:06:47.153 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-21T04:06:48.035 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:48.051 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-21T04:06:48.112 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:48.112 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:48.112 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image11 2023-10-21T04:06:48.160 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-21T04:06:49.040 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:49.053 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-21T04:06:49.114 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:49.115 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:49.115 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image12 2023-10-21T04:06:49.162 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-21T04:06:50.052 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:50.065 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-21T04:06:50.127 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:50.127 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:50.127 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image13 2023-10-21T04:06:50.174 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-21T04:06:51.078 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:51.090 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-21T04:06:51.152 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:51.152 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:51.153 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image14 2023-10-21T04:06:51.203 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-21T04:06:52.075 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:52.089 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-21T04:06:52.151 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:52.151 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:52.151 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image15 2023-10-21T04:06:52.201 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-21T04:06:53.077 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:53.088 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-21T04:06:53.154 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:53.154 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:53.154 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image16 2023-10-21T04:06:53.200 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-21T04:06:54.089 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:54.103 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-21T04:06:54.165 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:54.165 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:54.165 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image17 2023-10-21T04:06:54.213 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-21T04:06:55.097 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:55.110 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-21T04:06:55.170 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:55.171 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:55.171 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image18 2023-10-21T04:06:55.222 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-21T04:06:56.118 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:56.129 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-21T04:06:56.191 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:56.191 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:56.191 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image19 2023-10-21T04:06:56.240 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-21T04:06:57.120 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:57.134 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-21T04:06:57.195 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:57.195 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:57.196 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image20 2023-10-21T04:06:57.244 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-21T04:06:58.133 INFO:tasks.workunit.client.0.smithi172.stdout:Mirroring enabled 2023-10-21T04:06:58.147 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-21T04:06:58.208 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:58.209 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:58.209 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i = 1 )) 2023-10-21T04:06:58.209 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:58.209 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image1 2023-10-21T04:06:58.412 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-21T04:06:58.412 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:58.412 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:58.412 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:58.413 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image2 2023-10-21T04:06:58.484 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-21T04:06:58.485 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:58.485 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:58.485 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:58.485 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image3 2023-10-21T04:06:58.555 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-21T04:06:58.556 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:58.556 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:58.556 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:58.556 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image4 2023-10-21T04:06:58.629 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-21T04:06:58.629 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:58.629 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:58.630 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:58.630 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image5 2023-10-21T04:06:58.703 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-21T04:06:58.704 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:58.704 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:58.704 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:58.704 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image6 2023-10-21T04:06:58.814 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-21T04:06:58.815 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:58.815 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:58.816 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:58.816 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image7 2023-10-21T04:06:58.960 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-21T04:06:58.960 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:58.960 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:58.960 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image8 2023-10-21T04:06:58.961 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:59.096 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-21T04:06:59.096 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:59.096 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:59.096 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:59.097 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image9 2023-10-21T04:06:59.260 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-21T04:06:59.260 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:59.261 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:59.261 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:59.261 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image10 2023-10-21T04:06:59.424 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-21T04:06:59.424 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:59.425 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:59.425 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:59.426 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image11 2023-10-21T04:06:59.641 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-21T04:06:59.642 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:59.642 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:59.642 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:59.643 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image12 2023-10-21T04:06:59.877 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-21T04:06:59.878 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:06:59.878 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:06:59.878 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image13 2023-10-21T04:06:59.878 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:07:00.083 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-21T04:07:00.083 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:07:00.083 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:07:00.083 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:07:00.084 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image14 2023-10-21T04:07:00.284 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-21T04:07:00.285 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:07:00.285 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:07:00.285 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image15 2023-10-21T04:07:00.286 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:07:00.496 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-21T04:07:00.497 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:07:00.497 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:07:00.497 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image16 2023-10-21T04:07:00.497 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:07:00.859 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-21T04:07:00.859 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:07:00.860 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:07:00.860 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:07:00.860 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image17 2023-10-21T04:07:01.181 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-21T04:07:01.182 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:07:01.182 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:07:01.182 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:07:01.182 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image18 2023-10-21T04:07:01.380 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-21T04:07:01.381 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:07:01.381 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:07:01.381 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image19 2023-10-21T04:07:01.381 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:07:01.706 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-21T04:07:01.707 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:07:01.707 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:07:01.707 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:07:01.708 INFO:tasks.workunit.client.0.smithi172.stderr:++ sudo rbd device map rbd/image20 2023-10-21T04:07:02.056 INFO:tasks.workunit.client.0.smithi172.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-21T04:07:02.056 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T04:07:02.056 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 20 )) 2023-10-21T04:07:02.056 INFO:tasks.workunit.client.0.smithi172.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:07:02.057 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:07:02.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861222 2023-10-21T04:07:02.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ END_TIME=1697864822 2023-10-21T04:07:02.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ PREV_CLIENT_ADDR= 2023-10-21T04:07:02.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR= 2023-10-21T04:07:02.059 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:02.059 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n '' ]] 2023-10-21T04:07:02.059 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:07:12.059 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:12.059 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:12.060 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:12.060 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:07:12.888 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/2863900873 2023-10-21T04:07:12.888 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:07:12.889 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861232 2023-10-21T04:07:12.889 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:12.890 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/2863900873 ]] 2023-10-21T04:07:12.890 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/2863900873 != '' ]] 2023-10-21T04:07:12.890 INFO:tasks.workunit.client.0.smithi172.stderr:+ ceph osd blocklist add 172.21.15.172:0/2863900873 2023-10-21T04:07:14.973 INFO:tasks.workunit.client.0.smithi172.stderr:blocklisting 172.21.15.172:0/2863900873 until 2023-10-21T05:07:14.202107+0000 (3600 sec) 2023-10-21T04:07:15.015 INFO:tasks.workunit.client.0.smithi172.stderr:+ ceph osd blocklist ls 2023-10-21T04:07:15.015 INFO:tasks.workunit.client.0.smithi172.stderr:+ grep -q 172.21.15.172:0/2863900873 2023-10-21T04:07:15.768 INFO:tasks.workunit.client.0.smithi172.stderr:listed 1 entries 2023-10-21T04:07:15.818 INFO:tasks.workunit.client.0.smithi172.stderr:+ PREV_CLIENT_ADDR=172.21.15.172:0/2863900873 2023-10-21T04:07:15.819 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:07:25.822 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:07:25.822 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:25.823 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:25.823 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:26.767 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/2863900873 2023-10-21T04:07:26.768 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:07:26.769 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861246 2023-10-21T04:07:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/2863900873 ]] 2023-10-21T04:07:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/2863900873 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\2\8\6\3\9\0\0\8\7\3 ]] 2023-10-21T04:07:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:07:31.291 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:07:31.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:07:31.291 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:07:31.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:07:31.291 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:07:31.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:07:31.292 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:07:31.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:07:31.292 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:07:31.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:07:31.292 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:07:31.292+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:07:31.292 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:07:31.292+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:07:31.293 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:07:31.292+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:07:31.293 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:07:31.292+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:07:31.293 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:07:31.292+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:07:36.783 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:07:36.783 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:36.785 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:36.799 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:37.843 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/2863900873 2023-10-21T04:07:37.843 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:07:37.844 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861257 2023-10-21T04:07:37.844 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:37.844 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/2863900873 ]] 2023-10-21T04:07:37.844 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/2863900873 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\2\8\6\3\9\0\0\8\7\3 ]] 2023-10-21T04:07:37.844 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:07:47.856 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:07:47.856 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:47.859 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:47.866 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:49.084 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/2863900873 2023-10-21T04:07:49.087 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:07:49.088 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861269 2023-10-21T04:07:49.088 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/2863900873 ]] 2023-10-21T04:07:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/2863900873 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\2\8\6\3\9\0\0\8\7\3 ]] 2023-10-21T04:07:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:07:59.093 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:07:59.093 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:59.094 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:59.100 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:00.006 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.004+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.006 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.004+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e7288e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.009 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.008+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.009 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.008+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e66bce80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.016 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.016+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.016 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.016+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e6f14e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.024 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.024+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.025 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.024+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e7289480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.028 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.028+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.028 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.028+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e22d2100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.031 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.032+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.032 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.032+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e22d2300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.041 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.040+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.042 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.040+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e22d2500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.052 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.052+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.052 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.052+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e22d2700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.061 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.060+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.061 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.060+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e22d2900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.074 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.072+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.075 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.072+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e22d2b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.080 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.080+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.081 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.080+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e22d2d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.087 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.084+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.087 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.084+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e22d2000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.108 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.108+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.108 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.108+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e22d2200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.126 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.124+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.126 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.124+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e7288780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.133 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.132+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.134 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.132+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e7288d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.143 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.140+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.143 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.140+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e7289480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.155 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.156+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.155 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.156+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e66bce00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.164 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.164+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.164 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.164+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e6f51f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.176 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.176+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.176 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.176+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e22ec080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.189 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.188+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.189 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:08:00.188+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e22ec280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.330 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/2863900873 2023-10-21T04:08:00.331 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:08:00.331 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861280 2023-10-21T04:08:00.331 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:00.332 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/2863900873 ]] 2023-10-21T04:08:00.332 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/2863900873 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\2\8\6\3\9\0\0\8\7\3 ]] 2023-10-21T04:08:00.332 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:08:05.314 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:08:05.312+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:08:10.339 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:08:10.340 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:10.342 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:10.360 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:11.495 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/2863900873 2023-10-21T04:08:11.499 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:08:11.500 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861291 2023-10-21T04:08:11.500 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:11.500 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/2863900873 ]] 2023-10-21T04:08:11.500 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/2863900873 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\2\8\6\3\9\0\0\8\7\3 ]] 2023-10-21T04:08:11.501 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:08:21.503 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:08:21.503 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:21.504 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:21.514 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:22.591 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/4214757225 2023-10-21T04:08:22.594 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:08:22.595 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861302 2023-10-21T04:08:22.596 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:22.596 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/4214757225 ]] 2023-10-21T04:08:22.596 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/4214757225 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\2\8\6\3\9\0\0\8\7\3 ]] 2023-10-21T04:08:22.596 INFO:tasks.workunit.client.0.smithi172.stderr:+ ceph osd blocklist add 172.21.15.172:0/4214757225 2023-10-21T04:08:24.956 INFO:tasks.workunit.client.0.smithi172.stderr:blocklisting 172.21.15.172:0/4214757225 until 2023-10-21T05:08:24.214305+0000 (3600 sec) 2023-10-21T04:08:25.010 INFO:tasks.workunit.client.0.smithi172.stderr:+ grep -q 172.21.15.172:0/4214757225 2023-10-21T04:08:25.012 INFO:tasks.workunit.client.0.smithi172.stderr:+ ceph osd blocklist ls 2023-10-21T04:08:25.887 INFO:tasks.workunit.client.0.smithi172.stderr:listed 2 entries 2023-10-21T04:08:25.950 INFO:tasks.workunit.client.0.smithi172.stderr:+ PREV_CLIENT_ADDR=172.21.15.172:0/4214757225 2023-10-21T04:08:25.951 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:08:35.957 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:08:35.958 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:35.959 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:35.962 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:37.068 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/4214757225 2023-10-21T04:08:37.069 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:08:37.069 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861317 2023-10-21T04:08:37.069 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:37.070 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/4214757225 ]] 2023-10-21T04:08:37.070 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/4214757225 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\4\2\1\4\7\5\7\2\2\5 ]] 2023-10-21T04:08:37.070 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:08:47.071 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:47.074 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:08:47.085 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:47.090 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:48.720 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/4214757225 2023-10-21T04:08:48.729 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:08:48.730 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861328 2023-10-21T04:08:48.730 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:48.730 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/4214757225 ]] 2023-10-21T04:08:48.730 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/4214757225 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\4\2\1\4\7\5\7\2\2\5 ]] 2023-10-21T04:08:48.731 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:08:55.625 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T04:08:55.625 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:08:55.625 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:08:55.625 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:08:55.626 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:08:55.626 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:08:55.626 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:08:55.626 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:08:55.627 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:08:55.627 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:08:55.627 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:08:55.627 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:08:55.628 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:08:55.628 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:08:55.623+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:08:58.742 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:08:58.742 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:58.748 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:58.772 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:59.276 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:08:59.275+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:08:59.966 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/4214757225 2023-10-21T04:08:59.966 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:08:59.967 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861339 2023-10-21T04:08:59.967 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:59.968 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/4214757225 ]] 2023-10-21T04:08:59.968 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/4214757225 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\4\2\1\4\7\5\7\2\2\5 ]] 2023-10-21T04:08:59.968 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:09:00.006 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.003+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.007 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.003+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e741a100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.009 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.007+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.009 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.007+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e741a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.012 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.011+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.013 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.011+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e7251380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.029 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.027+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.029 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.027+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e66bd480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.042 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.039+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.042 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.039+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e7251200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.048 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.047+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.048 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.047+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e7251680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.063 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.063+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.063 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.063+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e741a200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.069 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.067+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.070 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.067+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e741a380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.073 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.071+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.074 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.071+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e741aa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.137 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.137 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.138 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.138 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.138 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.138 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.138 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e741a880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.139 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.139 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e741a680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.139 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e7251600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.140 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.140 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e7250700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.140 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e7251480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.140 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.140 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e7251700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.141 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e741a400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.141 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e741a000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.141 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e741aa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.141 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.141 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.135+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e741b380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.142 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.139+0000 7fa7c6e65640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.142 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:00.139+0000 7fa7c6e65640 -1 librbd::ImageState: 0x55a5e741a880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:09.975 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:09:09.979 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:09.982 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:09.983 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:10.761 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:09:10.759+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:09:11.219 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/4214757225 2023-10-21T04:09:11.220 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:09:11.222 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861351 2023-10-21T04:09:11.222 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:11.222 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/4214757225 ]] 2023-10-21T04:09:11.223 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/4214757225 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\4\2\1\4\7\5\7\2\2\5 ]] 2023-10-21T04:09:11.223 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:09:21.228 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:09:21.228 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:21.228 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:21.232 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:22.288 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/4214757225 2023-10-21T04:09:22.288 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:09:22.289 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861362 2023-10-21T04:09:22.289 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:22.289 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/4214757225 ]] 2023-10-21T04:09:22.289 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/4214757225 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\4\2\1\4\7\5\7\2\2\5 ]] 2023-10-21T04:09:22.290 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:09:25.627 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.618+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:09:25.628 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.626+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:09:25.628 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.626+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:09:25.628 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.626+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:09:25.629 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.626+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:09:25.629 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.626+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:09:25.629 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.626+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:09:25.629 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.626+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:09:25.629 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.626+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:09:25.630 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.626+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:09:25.630 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.626+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:09:25.630 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.626+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:09:25.630 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:09:25.626+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:09:32.295 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:32.299 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:32.301 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:32.310 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:09:33.578 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/1931795673 2023-10-21T04:09:33.579 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:09:33.579 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861373 2023-10-21T04:09:33.579 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:33.580 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/1931795673 ]] 2023-10-21T04:09:33.580 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/1931795673 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\4\2\1\4\7\5\7\2\2\5 ]] 2023-10-21T04:09:33.580 INFO:tasks.workunit.client.0.smithi172.stderr:+ ceph osd blocklist add 172.21.15.172:0/1931795673 2023-10-21T04:09:35.935 INFO:tasks.workunit.client.0.smithi172.stderr:blocklisting 172.21.15.172:0/1931795673 until 2023-10-21T05:09:35.221676+0000 (3600 sec) 2023-10-21T04:09:35.985 INFO:tasks.workunit.client.0.smithi172.stderr:+ ceph osd blocklist ls 2023-10-21T04:09:35.986 INFO:tasks.workunit.client.0.smithi172.stderr:+ grep -q 172.21.15.172:0/1931795673 2023-10-21T04:09:37.185 INFO:tasks.workunit.client.0.smithi172.stderr:listed 3 entries 2023-10-21T04:09:37.238 INFO:tasks.workunit.client.0.smithi172.stderr:+ PREV_CLIENT_ADDR=172.21.15.172:0/1931795673 2023-10-21T04:09:37.238 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:09:47.246 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:47.248 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:47.250 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:47.254 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:09:48.532 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/1931795673 2023-10-21T04:09:48.533 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:09:48.535 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861388 2023-10-21T04:09:48.535 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:48.535 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/1931795673 ]] 2023-10-21T04:09:48.536 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/1931795673 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\1\9\3\1\7\9\5\6\7\3 ]] 2023-10-21T04:09:48.536 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:09:58.540 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:09:58.541 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:58.542 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:58.547 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:59.653 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/1931795673 2023-10-21T04:09:59.653 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:09:59.654 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861399 2023-10-21T04:09:59.654 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:59.654 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/1931795673 ]] 2023-10-21T04:09:59.654 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/1931795673 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\1\9\3\1\7\9\5\6\7\3 ]] 2023-10-21T04:09:59.655 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:10:00.001 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:59.998+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.002 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:09:59.998+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e7251c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.007 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.006+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.007 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.006+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e6f50200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.015 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.014+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.016 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.014+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e22ecf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.020 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.018+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.021 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.018+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e6f51f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.027 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.026+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.028 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.026+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e741b580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.032 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.030+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.032 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.030+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e741a480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.045 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.042+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.045 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.042+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e741be80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.060 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.058+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.060 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.058+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e741bf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.077 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.074+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.077 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.074+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e7251e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.085 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.082+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.085 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.082+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e741a600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.089 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.086+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.090 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.086+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e7250800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.102 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.098+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.102 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.102+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e6f51f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.112 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.110+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.113 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.110+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e741b480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.138 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.134+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.138 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.134+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e741b580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.149 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.146+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.149 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.146+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e741a480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.169 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.166+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.169 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.166+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e76ce000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.176 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.174+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.177 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.174+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e76ce200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.183 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.182+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.183 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.182+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e76ce400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.186 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.186+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.186 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.186+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e76ced80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.192 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.190+0000 7fa7c7666640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.193 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:00.190+0000 7fa7c7666640 -1 librbd::ImageState: 0x55a5e76cec00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:09.158 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.154+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:10:09.162 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.158+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:10:09.162 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.158+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:10:09.181 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.174+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:10:09.186 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.174+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:10:09.187 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.174+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:10:09.187 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.174+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:10:09.187 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.174+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:10:09.187 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.174+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:10:09.188 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.174+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:10:09.188 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.174+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:10:09.188 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.174+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:10:09.188 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:10:09.174+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:10:09.655 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:10:09.656 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:09.656 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:09.658 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/1931795673 2023-10-21T04:10:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:10:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861410 2023-10-21T04:10:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/1931795673 ]] 2023-10-21T04:10:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/1931795673 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\1\9\3\1\7\9\5\6\7\3 ]] 2023-10-21T04:10:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:10:20.955 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:20.959 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:20.962 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:10:20.982 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:22.326 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/1931795673 2023-10-21T04:10:22.327 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:10:22.334 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861422 2023-10-21T04:10:22.334 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:22.334 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/1931795673 ]] 2023-10-21T04:10:22.334 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/1931795673 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\1\9\3\1\7\9\5\6\7\3 ]] 2023-10-21T04:10:22.334 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:10:32.347 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:32.347 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:32.348 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:10:32.348 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:33.498 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:10:33.499 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:10:33.503 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861433 2023-10-21T04:10:33.503 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:33.503 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:10:33.504 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\1\9\3\1\7\9\5\6\7\3 ]] 2023-10-21T04:10:33.504 INFO:tasks.workunit.client.0.smithi172.stderr:+ ceph osd blocklist add 172.21.15.172:0/3003149756 2023-10-21T04:10:35.968 INFO:tasks.workunit.client.0.smithi172.stderr:blocklisting 172.21.15.172:0/3003149756 until 2023-10-21T05:10:35.298117+0000 (3600 sec) 2023-10-21T04:10:36.018 INFO:tasks.workunit.client.0.smithi172.stderr:+ ceph osd blocklist ls 2023-10-21T04:10:36.020 INFO:tasks.workunit.client.0.smithi172.stderr:+ grep -q 172.21.15.172:0/3003149756 2023-10-21T04:10:37.243 INFO:tasks.workunit.client.0.smithi172.stderr:listed 4 entries 2023-10-21T04:10:37.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ PREV_CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:10:37.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:10:47.318 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:10:47.334 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:47.338 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:47.343 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:48.582 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:10:48.583 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:10:48.583 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861448 2023-10-21T04:10:48.583 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:48.584 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:10:48.584 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:10:48.584 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:10:58.595 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:10:58.596 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:58.603 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:58.613 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:59.910 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:10:59.910 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:10:59.911 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861459 2023-10-21T04:10:59.911 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:59.911 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:10:59.912 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:10:59.912 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:11:00.002 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:59.997+0000 7fa7c6664640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.002 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:10:59.997+0000 7fa7c6664640 -1 librbd::ImageState: 0x55a5e7508580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.009 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:11:00.005+0000 7fa7c6664640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.009 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:11:00.005+0000 7fa7c6664640 -1 librbd::ImageState: 0x55a5e7509580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.017 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:11:00.013+0000 7fa7c6664640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.017 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:11:00.013+0000 7fa7c6664640 -1 librbd::ImageState: 0x55a5e7509780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.033 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:11:00.033+0000 7fa7c6664640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.034 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T04:11:00.033+0000 7fa7c6664640 -1 librbd::ImageState: 0x55a5e7509980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:07.088 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:07.081+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:11:07.089 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:07.085+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:11:07.090 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:07.085+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:11:07.090 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:07.085+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:11:07.090 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:07.085+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:11:07.090 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:07.085+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:11:07.090 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:07.085+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:11:07.091 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:07.085+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:11:07.091 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:07.085+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:11:07.091 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:07.085+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:11:09.913 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:09.917 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:09.919 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:11:09.923 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:11.134 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:11:11.134 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:11:11.135 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861471 2023-10-21T04:11:11.135 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:11.135 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:11:11.135 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:11:11.136 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:11:16.245 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:16.237+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:11:16.246 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:16.237+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:11:16.246 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:16.237+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:11:16.246 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:16.237+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:11:16.246 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:16.237+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:11:16.247 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:16.237+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:11:16.247 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:16.237+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:11:16.247 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:16.237+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:11:16.247 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:16.237+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:11:16.247 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:11:16.237+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:11:21.140 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:11:21.141 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:21.141 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:21.152 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:22.368 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:11:22.369 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:11:22.372 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861482 2023-10-21T04:11:22.373 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:22.373 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:11:22.373 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:11:22.373 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:11:32.380 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:32.382 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:11:32.385 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:32.389 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:33.608 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:11:33.610 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:11:33.617 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861493 2023-10-21T04:11:33.618 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:33.618 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:11:33.618 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:11:33.618 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:11:43.620 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:11:43.621 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:43.621 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:43.628 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:44.949 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:11:44.950 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:11:44.957 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861504 2023-10-21T04:11:44.957 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:44.958 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:11:44.958 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:11:44.958 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:11:54.960 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:11:54.970 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:54.973 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:54.993 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:56.228 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:11:56.229 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:11:56.230 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861516 2023-10-21T04:11:56.231 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:56.231 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:11:56.231 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:11:56.231 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:12:06.237 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:06.237 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:12:06.244 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:06.250 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:07.314 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:12:07.314 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:12:07.315 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861527 2023-10-21T04:12:07.315 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:07.315 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:12:07.315 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:12:07.315 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:12:17.334 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:17.337 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:17.338 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:17.347 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:12:18.482 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:12:18.487 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:12:18.488 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861538 2023-10-21T04:12:18.488 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:18.488 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:12:18.489 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:12:18.489 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:12:28.491 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:28.491 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:12:28.494 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:28.501 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:29.803 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:12:29.805 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:12:29.806 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861549 2023-10-21T04:12:29.806 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:29.806 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:12:29.806 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:12:29.807 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:12:39.815 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:39.815 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:12:39.820 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:39.840 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:41.070 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:12:41.070 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:12:41.071 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861561 2023-10-21T04:12:41.072 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:41.072 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:12:41.072 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:12:41.072 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:12:47.452 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:12:47.451+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:12:51.074 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:12:51.075 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:51.079 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:51.080 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:52.209 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:12:52.210 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:12:52.211 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861572 2023-10-21T04:12:52.211 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:52.211 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:12:52.211 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:12:52.212 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:13:02.213 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:02.214 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:02.216 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:13:02.216 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:03.456 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:13:03.468 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:13:03.471 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861583 2023-10-21T04:13:03.471 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:03.471 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:13:03.472 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:13:03.472 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:13:13.495 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:13:13.496 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:13.497 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:13.501 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:14.719 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:13:14.720 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:13:14.720 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861594 2023-10-21T04:13:14.720 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:14.721 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:13:14.721 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:13:14.721 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:13:24.724 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:13:24.725 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:24.747 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:24.750 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:25.846 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:13:25.846 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:13:25.847 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861605 2023-10-21T04:13:25.847 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:25.847 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:13:25.847 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:13:25.847 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:13:27.915 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:13:27.914+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:13:27.920 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:13:27.918+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:13:27.921 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:13:27.918+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:13:27.921 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:13:27.918+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:13:27.921 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:13:27.918+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:13:27.921 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:13:27.918+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:13:27.922 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:13:27.918+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:13:27.922 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:13:27.918+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:13:27.922 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:13:27.918+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:13:27.923 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:13:27.918+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:13:35.850 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:13:35.850 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:35.850 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:35.851 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:37.135 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:13:37.135 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:13:37.136 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861617 2023-10-21T04:13:37.136 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:37.136 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:13:37.136 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:13:37.136 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:13:47.138 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:47.139 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:47.139 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:47.143 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:13:48.233 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:13:48.238 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:13:48.239 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861628 2023-10-21T04:13:48.239 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:48.239 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:13:48.240 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:13:48.240 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:13:49.930 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:13:49.930+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:13:58.245 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:58.247 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:13:58.262 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:58.263 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:59.281 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:13:59.286 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:13:59.288 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861639 2023-10-21T04:13:59.288 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:59.288 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:13:59.288 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:13:59.289 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:14:09.292 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:09.293 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:09.300 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:14:09.320 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:10.525 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:14:10.525 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:14:10.526 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861650 2023-10-21T04:14:10.526 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:10.526 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:14:10.526 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:14:10.527 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:14:20.529 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:14:20.530 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:20.532 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:20.535 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:21.637 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:14:21.638 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:14:21.638 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861661 2023-10-21T04:14:21.638 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:21.639 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:14:21.639 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:14:21.639 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:14:31.640 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:31.643 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:31.644 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:14:31.668 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:32.862 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:14:32.863 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:14:32.864 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861672 2023-10-21T04:14:32.864 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:32.865 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:14:32.865 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:14:32.865 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:14:42.870 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:14:42.871 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:42.871 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:42.871 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:43.818 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:14:43.818 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:14:43.819 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861683 2023-10-21T04:14:43.819 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:43.819 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:14:43.820 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:14:43.820 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:14:53.830 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:14:53.831 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:53.849 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:53.852 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:54.781 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:14:54.781+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:14:54.782 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:14:54.781+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:14:54.952 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:14:54.953 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:14:54.953 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861694 2023-10-21T04:14:54.953 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:54.954 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:14:54.954 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:14:54.954 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:14:58.549 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:14:58.549+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:15:04.955 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:15:04.956 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:04.958 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:04.960 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:06.313 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:15:06.313 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:15:06.314 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861706 2023-10-21T04:15:06.314 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:06.314 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:15:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:15:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:15:16.316 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:15:16.317 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:16.334 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:16.347 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:17.650 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:15:17.651 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:15:17.651 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861717 2023-10-21T04:15:17.651 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:17.652 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:15:17.652 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:15:17.652 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:15:27.653 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:15:27.654 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:27.654 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:27.655 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:28.792 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:15:28.796 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:15:28.797 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861728 2023-10-21T04:15:28.797 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:28.797 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:15:28.798 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:15:28.798 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:15:38.801 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:38.805 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:15:38.806 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:38.806 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:40.111 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:15:40.117 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:15:40.121 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861740 2023-10-21T04:15:40.122 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:40.122 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:15:40.122 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:15:40.122 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:15:50.126 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:15:50.127 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:50.139 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:50.149 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:51.508 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:15:51.514 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:15:51.515 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861751 2023-10-21T04:15:51.515 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:51.516 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:15:51.516 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:15:51.516 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:16:01.517 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:16:01.517 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:01.517 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:01.527 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:02.560 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:16:02.564+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:16:02.907 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:16:02.908 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:16:02.910 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861762 2023-10-21T04:16:02.910 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:02.911 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:16:02.911 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:16:02.911 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:16:12.912 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:16:12.913 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:12.914 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:12.920 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:14.090 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:16:14.090 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:16:14.091 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861774 2023-10-21T04:16:14.091 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:14.091 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:16:14.091 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:16:14.092 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:16:24.094 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:16:24.094 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:24.096 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:24.102 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:25.324 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:16:25.325 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:16:25.326 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861785 2023-10-21T04:16:25.326 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:25.326 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:16:25.326 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:16:25.327 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:16:30.816 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:16:30.815+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:16:30.817 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:16:30.815+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:16:35.334 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:35.335 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:35.336 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:35.341 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:16:36.540 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:16:36.540 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:16:36.541 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861796 2023-10-21T04:16:36.541 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:36.541 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:16:36.542 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:16:36.542 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:16:46.543 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:46.543 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:16:46.545 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:46.545 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:47.716 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:16:47.717 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:16:47.718 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861807 2023-10-21T04:16:47.718 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:47.718 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:16:47.718 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:16:47.718 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:16:57.719 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:57.720 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:57.722 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:16:57.727 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:58.858 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:16:58.859 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:16:58.859 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861818 2023-10-21T04:16:58.859 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:58.860 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:16:58.860 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:16:58.860 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:17:08.870 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:17:08.871 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:08.876 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:08.897 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:10.519 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:17:10.519 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:17:10.520 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861830 2023-10-21T04:17:10.520 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:10.520 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:17:10.520 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:17:10.521 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:17:20.524 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:20.525 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:20.525 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:17:20.526 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:21.737 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:17:21.741 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:17:21.742 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861841 2023-10-21T04:17:21.742 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:21.743 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:17:21.743 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:17:21.743 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:17:31.754 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:17:31.755 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:31.760 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:31.774 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:32.727 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:17:32.727 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:17:32.728 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861852 2023-10-21T04:17:32.728 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:32.728 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:17:32.728 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:17:32.729 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:17:33.354 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T04:17:33.354 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:17:33.355 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:17:33.355 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:17:33.355 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:17:33.355 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:17:33.355 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:17:33.355 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:17:33.356 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:17:33.356 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:17:33.356 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:17:33.356 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:17:33.356 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:17:33.357 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:17:33.355+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:17:42.730 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:17:42.731 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:42.733 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:42.769 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:43.863 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:17:43.864 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:17:43.865 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861863 2023-10-21T04:17:43.865 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:43.865 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:17:43.865 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:17:43.865 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:17:44.666 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:17:44.666+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:17:53.867 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:53.869 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:53.877 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:53.879 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:17:55.023 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:17:55.026 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:17:55.040 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861875 2023-10-21T04:17:55.040 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:55.040 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:17:55.040 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:17:55.041 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:18:05.043 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:18:05.044 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:05.056 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:05.059 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:06.141 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:18:06.147 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:18:06.161 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861886 2023-10-21T04:18:06.162 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:06.162 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:18:06.162 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:18:06.162 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:18:16.164 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:18:16.166 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:16.171 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:16.174 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:17.331 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:18:17.332 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:18:17.332 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861897 2023-10-21T04:18:17.332 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:17.332 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:18:17.333 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:18:17.333 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:18:25.191 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:18:25.194+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:18:25.193 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:18:25.194+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:18:25.193 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:18:25.194+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:18:25.193 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:18:25.194+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:18:25.193 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:18:25.194+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:18:25.194 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:18:25.194+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:18:25.194 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:18:25.194+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:18:25.206 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:18:25.210+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:18:25.206 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:18:25.210+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:18:25.207 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:18:25.210+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:18:27.334 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:18:27.334 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:27.335 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:27.338 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:27.546 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:18:27.546+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:18:28.607 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:18:28.613 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:18:28.614 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861908 2023-10-21T04:18:28.614 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:28.614 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:18:28.615 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:18:28.615 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:18:38.633 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:38.633 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:38.633 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:18:38.638 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:39.935 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:18:39.941 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:18:39.942 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861919 2023-10-21T04:18:39.942 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:39.942 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:18:39.943 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:18:39.943 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:18:49.954 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:18:49.955 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:49.958 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:49.958 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:51.141 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:18:51.142 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:18:51.269 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861931 2023-10-21T04:18:51.269 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:51.270 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:18:51.270 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:18:51.270 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:19:01.529 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:01.533 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:19:01.539 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:01.552 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:02.976 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:19:02.977 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:19:02.983 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861942 2023-10-21T04:19:02.983 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:02.984 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:19:02.984 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:19:02.984 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:19:13.015 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:13.021 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:13.021 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:19:13.022 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:14.634 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:19:14.634 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:19:14.635 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861954 2023-10-21T04:19:14.635 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:14.635 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:19:14.635 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:19:14.636 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:19:24.638 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:24.640 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:19:24.641 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:24.653 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:25.733 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:19:25.734 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:19:25.737 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861965 2023-10-21T04:19:25.737 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:25.737 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:19:25.737 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:19:25.738 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:19:35.746 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:19:35.749 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:35.749 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:35.763 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:36.850 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:19:36.850 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:19:36.851 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861976 2023-10-21T04:19:36.851 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:36.851 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:19:36.851 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:19:36.851 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:19:46.863 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:46.865 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:46.865 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:46.866 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:19:48.157 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:19:48.157 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:19:48.158 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861988 2023-10-21T04:19:48.159 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:48.159 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:19:48.159 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:19:48.159 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:19:58.173 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:19:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:58.177 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:58.190 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:59.487 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:19:59.488 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:19:59.488 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697861999 2023-10-21T04:19:59.489 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:59.489 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:19:59.489 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:19:59.489 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:20:09.491 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:20:09.492 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:09.492 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:09.497 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:10.742 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:20:10.742 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:20:10.742 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862010 2023-10-21T04:20:10.743 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:10.743 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:20:10.743 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:20:10.743 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:20:20.746 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:20:20.747 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:20.747 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:20.788 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:21.988 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:20:21.998 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:20:22.005 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862022 2023-10-21T04:20:22.005 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:22.005 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:20:22.005 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:20:22.006 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:20:32.006 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:32.007 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:32.007 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:32.012 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:20:32.961 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:20:32.962 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:20:32.962 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862032 2023-10-21T04:20:32.962 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:32.962 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:20:32.963 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:20:32.963 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:20:36.343 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:20:36.348+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:20:42.963 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:42.970 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:42.970 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:42.972 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:20:44.042 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:20:44.042 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:20:44.045 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862044 2023-10-21T04:20:44.045 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:44.045 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:20:44.045 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:20:44.045 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:20:54.051 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:54.053 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:20:54.054 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:54.086 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:55.353 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:20:55.353 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:20:55.355 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862055 2023-10-21T04:20:55.355 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:55.355 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:20:55.355 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:20:55.356 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:20:59.926 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:20:59.927+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:20:59.926 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:20:59.931+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:21:05.356 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:21:05.357 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:05.357 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:05.369 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:06.588 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:21:06.589 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:21:06.589 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862066 2023-10-21T04:21:06.590 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:06.590 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:21:06.590 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:21:06.590 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:21:16.600 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:21:16.609 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:16.612 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:16.630 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:17.642 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:21:17.643 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:21:17.645 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862077 2023-10-21T04:21:17.645 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:17.646 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:21:17.646 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:21:17.646 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:21:27.648 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:21:27.648 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:27.649 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:27.655 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:28.706 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:21:28.707 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:21:28.708 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862088 2023-10-21T04:21:28.708 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:28.709 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:21:28.709 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:21:28.709 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:21:38.713 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:21:38.713 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:38.719 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:38.724 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:39.942 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:21:39.945 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:21:39.958 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862099 2023-10-21T04:21:39.958 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:39.959 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:21:39.959 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:21:39.959 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:21:49.967 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:49.968 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:49.976 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:21:49.976 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:51.331 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:21:51.331 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:21:51.333 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862111 2023-10-21T04:21:51.334 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:51.334 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:21:51.334 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:21:51.334 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:22:01.339 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:01.339 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:22:01.342 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:01.342 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:02.400 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:22:02.400 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:22:02.400 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862122 2023-10-21T04:22:02.401 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:02.401 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:22:02.401 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:22:02.401 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:22:12.403 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:12.405 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:12.407 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:12.415 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:22:13.684 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:22:13.685 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:22:13.686 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862133 2023-10-21T04:22:13.687 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:13.687 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:22:13.687 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:22:13.687 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:22:22.902 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:22:22.893+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:22:22.903 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:22:22.901+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:22:22.903 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:22:22.901+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:22:22.904 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:22:22.901+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:22:23.710 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:22:23.711 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:23.711 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:23.716 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:24.782 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:22:24.782 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:22:24.783 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862144 2023-10-21T04:22:24.783 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:24.783 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:22:24.784 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:22:24.784 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:22:34.785 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:22:34.786 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:34.786 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:34.787 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:36.038 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:22:36.038 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:22:36.040 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862156 2023-10-21T04:22:36.040 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:36.040 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:22:36.040 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:22:36.041 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:22:46.042 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:46.043 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:22:46.043 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:46.044 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:47.178 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:22:47.178 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:22:47.179 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862167 2023-10-21T04:22:47.179 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:47.179 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:22:47.179 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:22:47.179 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:22:57.182 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:57.187 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:57.189 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:57.193 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:22:58.271 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:22:58.272 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:22:58.272 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862178 2023-10-21T04:22:58.273 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:58.273 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:22:58.273 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:22:58.273 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:23:08.281 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:08.282 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:08.284 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:23:08.294 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:09.497 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:23:09.497 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:23:09.498 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862189 2023-10-21T04:23:09.498 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:09.498 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:23:09.498 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:23:09.499 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:23:19.502 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:23:19.506 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:19.508 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:19.515 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:20.870 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:23:20.876 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:23:20.877 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862200 2023-10-21T04:23:20.877 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:20.878 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:23:20.878 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:23:20.878 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:23:30.892 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:23:30.892 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:30.893 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:30.893 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:31.971 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:23:31.971 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:23:31.972 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862211 2023-10-21T04:23:31.972 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:31.972 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:23:31.972 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:23:31.972 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:23:41.984 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:23:41.991 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:42.005 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:42.005 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:43.200 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:23:43.201 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:23:43.217 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862223 2023-10-21T04:23:43.218 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:43.218 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:23:43.218 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:23:43.219 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:23:50.633 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:23:50.630+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:23:50.634 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:23:50.634+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:23:50.634 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:23:50.634+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:23:50.634 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:23:50.634+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:23:50.634 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:23:50.634+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:23:50.635 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:23:50.634+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:23:50.635 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:23:50.634+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:23:53.220 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:53.221 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:23:53.222 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:53.226 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:54.392 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:23:54.392 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:23:54.393 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862234 2023-10-21T04:23:54.393 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:54.393 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:23:54.393 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:23:54.394 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:24:04.396 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:04.400 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:04.401 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:24:04.401 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:05.993 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:24:05.997 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:24:06.000 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862246 2023-10-21T04:24:06.000 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:06.000 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:24:06.001 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:24:06.001 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:24:16.002 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:24:16.011 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:16.015 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:16.025 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:17.408 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:24:17.410 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:24:17.411 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862257 2023-10-21T04:24:17.411 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:17.411 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:24:17.411 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:24:17.412 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:24:27.419 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:27.423 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:27.423 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:24:27.443 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:28.508 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:24:28.509 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:24:28.509 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862268 2023-10-21T04:24:28.509 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:28.510 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:24:28.510 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:24:28.510 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:24:38.517 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:24:38.517 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:38.518 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:38.526 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:39.700 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:24:39.700 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:24:39.700 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862279 2023-10-21T04:24:39.701 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:39.701 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:24:39.701 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:24:39.701 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:24:48.768 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:24:48.768+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:24:49.705 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:49.705 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:49.707 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:24:49.712 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:50.910 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:24:50.910 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:24:50.911 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862290 2023-10-21T04:24:50.911 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:50.911 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:24:50.912 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:24:50.912 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:25:00.939 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:00.947 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:25:00.957 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:00.990 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:02.047 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:25:02.047 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:25:02.048 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862302 2023-10-21T04:25:02.048 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:02.048 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:25:02.048 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:25:02.049 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:25:12.052 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:25:12.053 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:12.055 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:12.059 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:13.291 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:25:13.292 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:25:13.292 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862313 2023-10-21T04:25:13.292 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:13.293 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:25:13.293 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:25:13.293 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:25:23.294 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:23.295 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:23.295 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:23.296 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:25:24.578 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:25:24.582 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:25:24.584 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862324 2023-10-21T04:25:24.584 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:24.584 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:25:24.584 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:25:24.584 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:25:34.590 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:34.594 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:34.597 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:34.606 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:25:36.020 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:25:36.021 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:25:36.021 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862336 2023-10-21T04:25:36.022 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:36.022 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:25:36.022 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:25:36.022 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:25:46.026 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:25:46.027 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:46.035 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:46.035 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:47.039 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:25:47.039 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:25:47.040 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862347 2023-10-21T04:25:47.040 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:47.041 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:25:47.041 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:25:47.041 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:25:50.322 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:25:50.321+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:25:50.323 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:25:50.321+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:25:57.043 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:57.043 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:57.045 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:25:57.046 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:58.282 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:25:58.284 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:25:58.290 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862358 2023-10-21T04:25:58.290 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:58.290 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:25:58.290 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:25:58.290 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:26:08.296 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:26:08.297 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:08.302 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:08.304 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:09.972 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:26:09.972 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:26:09.980 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862369 2023-10-21T04:26:09.980 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:09.980 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:26:09.981 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:26:09.981 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:26:12.979 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T04:26:12.979 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:26:12.979 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:26:12.979 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:26:12.980 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:26:12.980 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:26:12.980 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:26:12.980 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:26:12.980 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:26:12.980 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:26:12.981 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:26:12.981 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:26:12.981 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:26:12.981 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:12.977+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:26:19.982 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:26:19.982 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:19.984 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:19.984 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:21.133 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:26:21.134 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:26:21.135 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862381 2023-10-21T04:26:21.135 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:21.135 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:26:21.136 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:26:21.136 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:26:31.137 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:26:31.138 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:31.140 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:31.140 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:31.471 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:31.472+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:26:31.472 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:31.472+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:26:31.472 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:31.472+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:26:31.472 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:31.472+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:26:31.472 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:31.472+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:26:31.472 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:31.472+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:26:31.473 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:31.472+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:26:31.473 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:31.472+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:26:31.473 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:31.472+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:26:31.473 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:26:31.472+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:26:32.199 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:26:32.200 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:26:32.200 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862392 2023-10-21T04:26:32.201 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:32.201 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:26:32.201 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:26:32.201 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:26:42.204 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:42.207 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:26:42.209 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:42.216 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:43.440 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:26:43.441 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:26:43.441 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862403 2023-10-21T04:26:43.441 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:43.442 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:26:43.442 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:26:43.442 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:26:53.443 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:26:53.444 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:53.446 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:53.447 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:54.418 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:26:54.419 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:26:54.420 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862414 2023-10-21T04:26:54.420 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:54.420 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:26:54.421 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:26:54.421 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:27:04.427 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:04.427 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:04.438 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:04.448 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:27:05.695 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:27:05.699 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:27:05.700 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862425 2023-10-21T04:27:05.700 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:27:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:27:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:27:15.705 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:27:15.705 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:15.716 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:15.722 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:17.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:27:17.060 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:27:17.062 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862437 2023-10-21T04:27:17.062 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:17.062 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:27:17.062 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:27:17.062 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:27:27.063 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:27:27.064 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:27.064 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:27.065 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:28.160 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:27:28.161 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:27:28.161 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862448 2023-10-21T04:27:28.162 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:28.162 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:27:28.162 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:27:28.162 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:27:38.167 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:38.168 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:38.168 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:38.173 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:27:38.787 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T04:27:38.787 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:27:38.788 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:27:38.788 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:27:38.788 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:27:38.788 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:27:38.788 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:27:38.789 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:27:38.789 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:27:38.789 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:27:38.789 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:27:38.789 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:27:38.789 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:27:38.790 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:38.782+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:27:39.415 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:27:39.416 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:27:39.417 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862459 2023-10-21T04:27:39.417 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:39.417 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:27:39.418 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:27:39.418 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:27:40.971 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:40.970+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:27:40.974 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:40.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:27:40.974 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:40.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:27:40.974 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:40.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:27:40.975 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:40.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:27:40.975 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:40.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:27:40.976 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:40.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:27:40.976 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:40.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:27:40.976 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:40.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:27:40.976 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:40.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:27:49.422 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:49.422 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:27:49.425 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:49.425 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:50.637 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:27:50.637 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:27:50.637 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862470 2023-10-21T04:27:50.638 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:50.638 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:27:50.638 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:27:50.638 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:27:53.044 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:53.042+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:27:53.045 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:53.042+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:27:53.046 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:53.042+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:27:53.052 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:53.046+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:27:53.052 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:53.046+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:27:53.053 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:53.046+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:27:53.053 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:53.046+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:27:53.053 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:53.046+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:27:53.054 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:53.046+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:27:53.054 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:53.046+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:27:53.054 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:53.046+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:27:53.055 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:27:53.046+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:28:00.664 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:00.664 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:00.665 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:28:00.665 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:01.889 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:28:01.890 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:28:01.891 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862481 2023-10-21T04:28:01.892 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:01.892 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:28:01.892 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:28:01.892 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:28:07.860 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:28:07.857+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:28:11.924 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:11.925 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:11.925 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:11.925 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:28:13.162 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:28:13.174 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:28:13.175 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862493 2023-10-21T04:28:13.176 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:13.176 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:28:13.176 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:28:13.176 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:28:23.181 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:23.190 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:28:23.192 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:23.207 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:24.441 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:28:24.442 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:28:24.445 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862504 2023-10-21T04:28:24.445 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:24.445 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:28:24.446 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:28:24.446 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:28:34.443 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:28:34.444 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:34.444 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:34.444 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:35.760 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:28:35.765 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:28:35.766 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862515 2023-10-21T04:28:35.766 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:35.767 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:28:35.767 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:28:35.767 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:28:45.768 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:45.773 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:45.776 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:45.784 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:28:47.075 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:28:47.082 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:28:47.083 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862527 2023-10-21T04:28:47.083 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:47.083 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:28:47.083 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:28:47.084 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:28:57.093 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:57.097 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:57.099 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:57.115 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:28:58.388 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:28:58.389 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:28:58.390 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862538 2023-10-21T04:28:58.390 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:58.390 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:28:58.390 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:28:58.391 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:29:08.426 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:08.428 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:29:08.432 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:08.446 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:09.697 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:29:09.701 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:29:09.708 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862549 2023-10-21T04:29:09.709 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:09.709 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:29:09.709 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:29:09.710 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:29:19.710 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:29:19.713 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:19.714 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:19.714 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:20.943 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:29:20.944 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:29:20.945 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862560 2023-10-21T04:29:20.945 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:20.945 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:29:20.945 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:29:20.946 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:29:30.947 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:29:30.948 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:30.948 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:30.949 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:32.213 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:29:32.213 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:29:32.214 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862572 2023-10-21T04:29:32.214 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:32.214 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:29:32.214 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:29:32.215 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:29:42.154 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:29:42.150+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:29:42.219 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:42.219 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:29:42.231 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:42.241 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:43.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:29:43.311 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:29:43.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862583 2023-10-21T04:29:43.312 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:43.312 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:29:43.312 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:29:43.312 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:29:53.316 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:53.317 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:53.318 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:29:53.327 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:54.467 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:29:54.471 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:29:54.473 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862594 2023-10-21T04:29:54.473 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:54.473 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:29:54.474 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:29:54.474 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:29:57.245 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:29:57.242+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:29:57.246 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:29:57.242+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:30:04.506 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:04.509 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:04.512 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:04.514 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:30:05.812 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:30:05.812 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:30:05.813 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862605 2023-10-21T04:30:05.813 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:05.813 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:30:05.813 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:30:05.814 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:30:09.158 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:30:09.159 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:30:09.159 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:30:09.159 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:30:09.159 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:30:09.159 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:30:09.160 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:30:09.160 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:30:09.160 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:30:09.160 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:30:09.160 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:30:09.161 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:30:09.161 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:30:09.157+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:30:15.821 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:30:15.821 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:15.822 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:15.831 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:17.178 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:30:17.179 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:30:17.180 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862617 2023-10-21T04:30:17.180 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:17.180 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:30:17.181 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:30:17.181 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:30:27.183 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:30:27.184 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:27.184 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:27.216 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:28.266 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:30:28.267 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:30:28.267 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862628 2023-10-21T04:30:28.267 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:28.268 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:30:28.268 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:30:28.268 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:30:38.269 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:30:38.270 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:38.270 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:38.271 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:39.541 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:30:39.541 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:30:39.542 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862639 2023-10-21T04:30:39.542 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:39.542 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:30:39.543 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:30:39.543 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:30:49.546 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:49.546 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:30:49.549 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:49.560 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:50.752 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:30:50.752 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:30:50.755 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862650 2023-10-21T04:30:50.755 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:50.755 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:30:50.755 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:30:50.755 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:31:00.763 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:00.764 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:31:00.764 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:00.778 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:01.872 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:31:01.873 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:31:01.873 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862661 2023-10-21T04:31:01.874 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:01.874 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:31:01.874 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:31:01.874 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:31:11.879 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:31:11.880 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:11.882 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:11.898 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:13.051 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:31:13.052 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:31:13.052 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862673 2023-10-21T04:31:13.053 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:13.053 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:31:13.053 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:31:13.053 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:31:23.068 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:31:23.068 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:23.070 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:23.083 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:24.176 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:31:24.176 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:31:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862684 2023-10-21T04:31:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:31:24.178 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:31:24.178 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:31:34.183 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:34.184 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:34.185 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:31:34.190 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:35.568 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:31:35.569 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:31:35.570 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862695 2023-10-21T04:31:35.571 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:35.571 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:31:35.571 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:31:35.571 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:31:45.573 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:31:45.573 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:45.583 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:45.601 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:46.852 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:31:46.852 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:31:46.853 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862706 2023-10-21T04:31:46.853 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:46.853 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:31:46.853 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:31:46.854 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:31:56.855 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:56.856 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:31:56.858 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:56.870 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:58.102 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:31:58.105 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:31:58.106 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862718 2023-10-21T04:31:58.106 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:58.107 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:31:58.107 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:31:58.107 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:32:08.116 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:08.116 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:08.116 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:08.117 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:32:09.518 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:32:09.520 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:32:09.521 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862729 2023-10-21T04:32:09.522 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:09.522 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:32:09.522 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:32:09.522 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:32:14.019 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:32:14.014+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:32:19.555 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:19.558 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:19.559 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:32:19.559 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:20.856 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:32:20.856 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:32:20.857 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862740 2023-10-21T04:32:20.857 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:32:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:32:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:32:30.859 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:32:30.860 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:30.860 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:30.869 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:32.141 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:32:32.143 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:32:32.144 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862752 2023-10-21T04:32:32.144 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:32.144 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:32:32.144 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:32:32.144 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:32:42.146 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:32:42.147 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:42.147 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:42.160 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:43.562 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:32:43.563 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:32:43.564 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862763 2023-10-21T04:32:43.565 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:43.565 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:32:43.565 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:32:43.565 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:32:53.577 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:53.579 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:53.583 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:53.587 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:32:54.728 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:32:54.728 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:32:54.729 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862774 2023-10-21T04:32:54.729 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:54.730 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:32:54.730 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:32:54.730 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:33:04.732 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:04.735 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:33:04.735 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:04.748 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:05.901 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:33:05.905 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:33:05.909 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862785 2023-10-21T04:33:05.909 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:05.910 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:33:05.910 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:33:05.910 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:33:10.569 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:33:10.568+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:33:15.913 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:33:15.913 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:15.913 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:15.914 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:16.971 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:33:16.972 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:33:16.973 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862796 2023-10-21T04:33:16.973 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:16.973 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:33:16.973 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:33:16.973 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:33:26.975 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:33:26.976 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:26.976 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:26.982 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:28.119 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:33:28.120 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:33:28.123 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862808 2023-10-21T04:33:28.123 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:28.123 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:33:28.123 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:33:28.123 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:33:38.125 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:38.126 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:33:38.131 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:38.145 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:39.233 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:33:39.234 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:33:39.236 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862819 2023-10-21T04:33:39.236 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:39.236 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:33:39.237 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:33:39.237 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:33:49.237 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:49.238 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:49.252 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:49.252 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:33:50.517 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:33:50.522 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:33:50.524 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862830 2023-10-21T04:33:50.524 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:50.524 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:33:50.525 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:33:50.525 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:34:00.530 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:34:00.530 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:00.530 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:00.534 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:01.751 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:34:01.752 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:34:01.753 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862841 2023-10-21T04:34:01.754 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:01.754 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:34:01.754 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:34:01.754 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:34:11.756 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:34:11.756 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:11.757 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:11.770 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:13.098 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:34:13.099 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:34:13.101 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862853 2023-10-21T04:34:13.101 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:13.101 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:34:13.101 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:34:13.102 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:34:23.110 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:34:23.110 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:23.111 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:23.112 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:24.309 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:34:24.309 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:34:24.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862864 2023-10-21T04:34:24.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:24.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:34:24.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:34:24.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:34:34.313 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:34:34.313 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:34.313 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:34.342 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:35.692 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:34:35.692 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:34:35.692 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862875 2023-10-21T04:34:35.693 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:35.693 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:34:35.693 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:34:35.693 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:34:45.693 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:45.693 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:34:45.699 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:45.699 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:46.865 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:34:46.865 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:34:46.866 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862886 2023-10-21T04:34:46.866 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:46.866 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:34:46.867 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:34:46.867 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:34:56.869 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:56.858+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:34:56.875 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:56.870+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:34:56.876 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:56.870+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:34:56.876 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:56.870+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:34:56.876 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:56.874+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:34:56.876 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:34:56.877 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:56.884 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:56.894 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:57.796 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T04:34:57.797 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:34:57.797 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:34:57.797 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:34:57.798 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:34:57.798 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:34:57.798 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:34:57.798 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:34:57.798 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:34:57.799 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:34:57.799 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:34:57.799 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:34:57.799 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:34:57.799 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:34:57.794+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:34:57.879 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:34:57.880 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:34:57.882 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862897 2023-10-21T04:34:57.883 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:57.883 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:34:57.883 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:34:57.883 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:35:01.976 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:01.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:35:01.977 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:01.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:35:01.977 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:01.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:35:01.978 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:01.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:35:01.978 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:01.974+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:35:07.889 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:07.892 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:35:07.893 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:07.893 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:09.136 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:35:09.136 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:35:09.137 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862909 2023-10-21T04:35:09.137 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:09.137 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:35:09.137 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:35:09.137 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:35:16.469 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:35:16.466+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:35:19.143 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:19.144 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:19.161 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:35:19.175 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:20.422 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:35:20.440 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:35:20.441 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862920 2023-10-21T04:35:20.441 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:20.442 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:35:20.442 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:35:20.442 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:35:30.107 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T04:35:30.107 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:35:30.107 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:35:30.107 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:35:30.107 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:35:30.108 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:35:30.108 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:35:30.108 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:35:30.108 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:35:30.108 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:35:30.109 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:35:30.109 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:35:30.109 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:35:30.109 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:35:30.101+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:35:30.443 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:30.448 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:30.448 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:30.454 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:35:31.693 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:35:31.694 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:35:31.695 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862931 2023-10-21T04:35:31.696 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:31.696 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:35:31.696 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:35:31.696 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:35:41.707 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:35:41.712 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:41.714 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:41.733 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:43.025 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:35:43.032 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:35:43.033 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862943 2023-10-21T04:35:43.033 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:43.034 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:35:43.034 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:35:43.034 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:35:53.038 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:53.042 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:53.046 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:35:53.074 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:54.136 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:35:54.136 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:35:54.137 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862954 2023-10-21T04:35:54.137 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:54.137 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:35:54.137 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:35:54.138 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:36:04.138 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:36:04.139 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:04.139 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:04.139 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:05.387 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:36:05.387 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:36:05.388 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862965 2023-10-21T04:36:05.388 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:05.389 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:36:05.389 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:36:05.389 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:36:15.390 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:36:15.391 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:15.391 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:15.422 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:16.672 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:36:16.673 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:36:16.674 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862976 2023-10-21T04:36:16.674 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:16.674 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:36:16.674 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:36:16.675 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:36:16.966 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:36:16.960+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:36:16.966 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:36:16.960+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:36:16.966 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:36:16.964+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:36:26.684 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:26.684 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:36:26.697 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:26.698 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:27.914 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:36:27.915 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:36:27.917 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862987 2023-10-21T04:36:27.917 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:27.917 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:36:27.917 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:36:27.918 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:36:30.673 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:36:30.672+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:36:37.918 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:36:37.919 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:37.919 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:37.929 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:38.761 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:36:38.765 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:36:38.766 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697862998 2023-10-21T04:36:38.767 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:38.767 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:36:38.767 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:36:38.767 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:36:48.791 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:36:48.791 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:48.798 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:48.804 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:49.905 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:36:49.911 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:36:49.912 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863009 2023-10-21T04:36:49.912 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:49.913 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:36:49.913 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:36:49.913 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:36:59.918 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:59.918 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:59.918 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:36:59.919 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:01.220 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:37:01.220 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:37:01.221 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863021 2023-10-21T04:37:01.221 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:01.221 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:37:01.221 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:37:01.222 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:37:11.223 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:37:11.223 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:11.225 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:11.251 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:12.425 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:37:12.425 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:37:12.435 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863032 2023-10-21T04:37:12.435 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:12.436 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:37:12.436 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:37:12.436 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:37:14.598 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:37:14.595+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:37:14.606 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:37:14.599+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:37:14.606 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:37:14.599+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:37:14.606 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:37:14.599+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:37:14.606 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:37:14.599+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:37:22.437 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:22.437 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:37:22.457 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:22.466 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:23.673 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:37:23.674 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:37:23.674 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863043 2023-10-21T04:37:23.675 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:23.675 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:37:23.675 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:37:23.675 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:37:33.680 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:37:33.681 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:33.681 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:33.681 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:35.033 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:37:35.033 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:37:35.035 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863055 2023-10-21T04:37:35.035 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:35.035 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:37:35.036 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:37:35.036 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:37:45.042 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:37:45.043 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:45.043 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:45.059 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:46.302 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:37:46.304 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:37:46.306 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863066 2023-10-21T04:37:46.306 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:46.306 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:37:46.306 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:37:46.307 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:37:56.317 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:56.320 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:56.320 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:37:56.322 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:57.635 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:37:57.636 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:37:57.638 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863077 2023-10-21T04:37:57.638 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:57.639 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:37:57.639 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:37:57.639 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:38:07.641 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:07.642 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:38:07.643 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:07.646 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:08.998 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:38:08.998 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:38:08.999 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863089 2023-10-21T04:38:09.000 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:09.000 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:38:09.000 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:38:09.000 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:38:19.003 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:19.003 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:38:19.009 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:19.012 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:20.370 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:38:20.371 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:38:20.371 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863100 2023-10-21T04:38:20.371 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:20.371 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:38:20.372 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:38:20.372 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:38:30.377 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:38:30.379 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:30.386 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:30.388 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:31.668 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:38:31.668 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:38:31.668 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863111 2023-10-21T04:38:31.669 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:31.669 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:38:31.669 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:38:31.669 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:38:41.674 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:41.674 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:41.675 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:38:41.677 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:42.958 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:38:42.959 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:38:42.959 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863122 2023-10-21T04:38:42.960 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:42.960 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:38:42.960 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:38:42.960 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:38:48.589 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:48.588+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:38:48.591 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:48.588+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:38:48.591 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:48.588+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:38:48.593 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:48.592+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:38:48.593 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:48.592+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:38:48.593 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:48.592+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:38:48.598 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:48.596+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:38:48.599 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:48.596+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:38:48.599 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:48.596+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:38:48.599 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:48.596+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:38:48.599 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:48.596+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:38:49.893 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:49.892+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:38:49.893 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:38:49.892+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:38:52.964 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:52.964 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:38:52.973 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:52.976 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:54.223 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:38:54.236 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:38:54.238 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863134 2023-10-21T04:38:54.238 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:54.239 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:38:54.239 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:38:54.239 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:39:04.240 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:04.245 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:04.246 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:04.251 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:39:05.640 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:39:05.640 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:39:05.641 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863145 2023-10-21T04:39:05.641 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:05.641 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:39:05.641 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:39:05.642 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:39:15.643 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:39:15.651 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:15.651 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:15.661 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:17.031 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:39:17.037 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:39:17.047 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863157 2023-10-21T04:39:17.047 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:17.047 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:39:17.048 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:39:17.048 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:39:27.059 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:27.061 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:27.062 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:39:27.069 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:28.346 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:39:28.346 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:39:28.346 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863168 2023-10-21T04:39:28.347 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:28.347 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:39:28.347 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:39:28.347 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:39:37.334 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T04:39:37.335 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:39:37.335 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:39:37.335 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:39:37.335 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:39:37.336 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:39:37.336 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:39:37.336 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:39:37.336 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:39:37.337 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:39:37.337 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:39:37.337 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:39:37.337 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:39:37.337 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:37.331+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:39:38.352 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:38.356 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:38.364 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:39:38.367 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:39.490 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:39:39.499 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:39:39.505 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863179 2023-10-21T04:39:39.505 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:39.505 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:39:39.506 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:39:39.506 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:39:49.518 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:39:49.519 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:49.519 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:49.526 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:50.814 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:39:50.815 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:39:50.816 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863190 2023-10-21T04:39:50.816 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:50.816 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:39:50.817 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:39:50.817 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:39:56.209 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:39:56.207+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:40:00.831 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:40:00.832 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:00.837 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:00.837 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:02.182 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:40:02.183 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:40:02.184 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863202 2023-10-21T04:40:02.184 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:02.184 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:40:02.184 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:40:02.184 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:40:12.193 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:40:12.193 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:12.194 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:12.202 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:13.100 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:40:13.100 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:40:13.101 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863213 2023-10-21T04:40:13.101 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:13.101 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:40:13.102 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:40:13.102 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:40:23.104 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:23.105 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:40:24.140 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:40:24.142 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:40:24.143 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863224 2023-10-21T04:40:24.143 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:24.143 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:40:24.143 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:40:24.144 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:40:34.152 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:34.154 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:34.155 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:34.193 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:40:35.332 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:40:35.332 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:40:35.334 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863235 2023-10-21T04:40:35.334 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:35.335 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:40:35.335 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:40:35.335 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:40:45.338 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:45.338 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:40:45.339 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:45.339 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:46.391 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:40:46.391 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:40:46.392 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863246 2023-10-21T04:40:46.392 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:46.392 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:40:46.393 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:40:46.393 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:40:49.234 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.229+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T04:40:49.234 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.229+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:40:49.236 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.233+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:40:49.236 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.233+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:40:49.237 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.233+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:40:49.237 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.233+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:40:49.237 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.233+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:40:49.237 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.233+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:40:49.237 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.233+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:40:49.238 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.233+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:40:49.238 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.233+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:40:49.238 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.233+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:40:49.238 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:40:49.233+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:40:56.394 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:40:56.394 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:56.394 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:56.395 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:57.649 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:40:57.649 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:40:57.660 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863257 2023-10-21T04:40:57.660 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:57.660 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:40:57.661 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:40:57.661 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:41:07.669 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:41:07.670 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:07.675 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:07.676 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:08.904 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:41:08.909 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:41:08.910 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863268 2023-10-21T04:41:08.910 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:08.911 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:41:08.911 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:41:08.911 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:41:18.912 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:41:18.913 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:18.914 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:18.921 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:20.010 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:41:20.010 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:41:20.011 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863280 2023-10-21T04:41:20.011 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:20.011 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:41:20.011 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:41:20.012 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:41:30.012 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:41:30.020 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:30.024 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:30.041 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:30.992 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:41:30.993 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:41:30.994 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863290 2023-10-21T04:41:30.994 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:30.994 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:41:30.994 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:41:30.994 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:41:40.999 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:41.003 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:41:41.007 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:41.008 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:42.355 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:41:42.361 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:41:42.362 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863302 2023-10-21T04:41:42.362 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:42.363 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:41:42.363 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:41:42.363 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:41:52.366 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:41:52.366 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:52.366 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:52.367 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:53.626 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:41:53.628 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:41:53.629 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863313 2023-10-21T04:41:53.629 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:53.629 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:41:53.630 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:41:53.630 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:42:03.636 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:42:03.636 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:03.637 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:03.638 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:05.088 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:42:05.089 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:42:05.090 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863325 2023-10-21T04:42:05.090 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:05.090 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:42:05.090 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:42:05.091 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:42:15.092 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:42:15.092 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:15.099 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:15.125 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:16.007 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:42:16.008 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:42:16.011 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863336 2023-10-21T04:42:16.011 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:16.011 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:42:16.012 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:42:16.012 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:42:26.014 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:42:26.015 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:26.024 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:26.032 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:27.241 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:42:27.241 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:42:27.242 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863347 2023-10-21T04:42:27.242 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:27.243 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:42:27.243 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:42:27.243 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:42:37.249 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:37.249 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:37.259 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:42:37.273 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:38.847 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:42:38.847 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:42:38.848 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863358 2023-10-21T04:42:38.848 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:38.848 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:42:38.849 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:42:38.849 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:42:41.117 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:42:41.111+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:42:41.118 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:42:41.115+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:42:41.125 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:42:41.123+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:42:41.125 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:42:41.123+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:42:41.125 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:42:41.123+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:42:41.126 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:42:41.123+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:42:41.126 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:42:41.123+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:42:41.126 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:42:41.123+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:42:41.126 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:42:41.123+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:42:41.127 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:42:41.123+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:42:41.127 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:42:41.123+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:42:48.851 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:42:48.851 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:48.851 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:48.852 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:50.117 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:42:50.118 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:42:50.118 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863370 2023-10-21T04:42:50.119 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:50.119 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:42:50.119 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:42:50.119 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:43:00.121 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:00.121 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:00.122 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:43:01.212 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:43:01.213 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:43:01.216 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863381 2023-10-21T04:43:01.216 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:01.216 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:43:01.216 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:43:01.216 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:43:11.221 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:11.227 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:11.237 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:11.244 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:43:12.466 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:43:12.481 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:43:12.482 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863392 2023-10-21T04:43:12.483 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:12.483 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:43:12.483 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:43:12.483 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:43:22.489 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:43:22.489 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:22.496 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:22.499 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:23.687 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:43:23.687 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:43:23.688 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863403 2023-10-21T04:43:23.688 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:23.689 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:43:23.689 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:43:23.689 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:43:33.701 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:33.708 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:43:33.719 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:33.729 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:35.004 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:43:35.005 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:43:35.005 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863415 2023-10-21T04:43:35.005 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:35.005 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:43:35.006 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:43:35.006 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:43:39.776 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:43:39.769+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:43:39.782 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:43:39.777+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:43:39.782 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:43:39.777+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:43:39.783 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:43:39.777+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:43:39.783 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:43:39.777+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:43:39.784 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:43:39.777+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:43:39.784 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:43:39.777+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:43:39.784 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:43:39.777+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:43:39.784 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:43:39.777+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:43:45.008 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:45.008 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:45.016 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:45.019 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:43:45.993 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:43:45.994 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:43:45.995 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863425 2023-10-21T04:43:45.995 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:45.995 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:43:45.995 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:43:45.996 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:43:55.999 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:43:56.000 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:56.002 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:56.002 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:57.562 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:43:57.562 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:43:57.564 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863437 2023-10-21T04:43:57.564 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:57.564 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:43:57.564 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:43:57.564 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:44:07.566 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:07.571 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:44:07.573 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:07.584 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:08.775 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:44:08.776 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:44:08.781 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863448 2023-10-21T04:44:08.782 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:08.782 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:44:08.782 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:44:08.782 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:44:16.190 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:44:16.185+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:44:18.408 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:44:18.405+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:44:18.803 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:18.808 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:44:18.808 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:18.823 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:19.807 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:44:19.815 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:44:19.816 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863459 2023-10-21T04:44:19.816 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:19.816 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:44:19.816 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:44:19.817 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:44:29.823 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:44:29.824 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:29.830 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:29.831 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:31.101 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:44:31.102 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:44:31.102 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863471 2023-10-21T04:44:31.103 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:31.103 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:44:31.103 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:44:31.103 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:44:41.111 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:44:41.111 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:41.113 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:41.133 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:42.242 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:44:42.242 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:44:42.243 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863482 2023-10-21T04:44:42.243 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:42.243 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:44:42.244 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:44:42.244 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:44:52.245 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:44:52.246 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:52.246 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:52.246 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:53.410 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:44:53.410 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:44:53.411 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863493 2023-10-21T04:44:53.411 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:53.411 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:44:53.412 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:44:53.412 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:45:03.413 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:45:03.413 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:03.414 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:03.417 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:04.644 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:45:04.645 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:45:04.652 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863504 2023-10-21T04:45:04.653 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:04.653 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:45:04.653 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:45:04.653 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:45:14.654 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:45:14.655 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:14.655 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:14.655 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:15.711 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:45:15.711 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:45:15.720 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863515 2023-10-21T04:45:15.721 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:15.721 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:45:15.721 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:45:15.721 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:45:25.723 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:25.723 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:45:25.730 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:25.733 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:26.883 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:45:26.883 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:45:26.884 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863526 2023-10-21T04:45:26.884 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:26.884 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:45:26.884 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:45:26.885 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:45:35.751 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T04:45:35.751 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:45:35.751 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:45:35.752 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:45:35.752 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:45:35.752 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:45:35.752 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:45:35.752 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:45:35.753 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:45:35.753 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:45:35.753 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:45:35.753 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:45:35.754 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:45:35.754 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:45:35.747+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:45:36.890 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:36.891 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:36.891 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:36.898 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:45:38.185 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:45:38.186 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:45:38.186 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863538 2023-10-21T04:45:38.187 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:38.187 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:45:38.187 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:45:38.187 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:45:48.204 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:45:48.205 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:48.213 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:48.221 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:49.367 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:45:49.368 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:45:49.368 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863549 2023-10-21T04:45:49.369 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:49.369 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:45:49.369 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:45:49.369 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:45:59.370 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:45:59.375 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:59.378 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:59.379 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:00.702 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:46:00.702 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:46:00.703 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863560 2023-10-21T04:46:00.703 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:00.703 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:46:00.704 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:46:00.704 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:46:10.709 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:10.710 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:10.713 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:10.721 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:46:11.834 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:46:11.834 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:46:11.836 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863571 2023-10-21T04:46:11.836 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:11.836 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:46:11.837 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:46:11.837 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:46:21.844 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:46:21.847 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:21.850 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:21.870 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:22.968 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:46:22.969 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:46:22.969 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863582 2023-10-21T04:46:22.969 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:22.970 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:46:22.970 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:46:22.970 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:46:32.978 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:32.978 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:32.979 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:46:32.979 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:34.353 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:46:34.357 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:46:34.359 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863594 2023-10-21T04:46:34.359 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:34.360 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:46:34.360 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:46:34.360 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:46:44.368 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:44.368 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:44.373 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:46:44.382 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:45.574 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:46:45.576 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:46:45.578 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863605 2023-10-21T04:46:45.578 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:45.578 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:46:45.579 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:46:45.579 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:46:55.580 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:46:55.581 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:55.582 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:55.586 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:56.966 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:46:56.975 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:46:56.978 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863616 2023-10-21T04:46:56.978 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:56.979 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:46:56.979 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:46:56.979 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:47:06.982 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:47:06.983 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:06.983 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:06.983 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:08.193 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:47:08.193 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:47:08.194 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863628 2023-10-21T04:47:08.194 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:08.195 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:47:08.195 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:47:08.195 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:47:18.196 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:18.198 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:18.199 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:47:18.203 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:19.308 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:47:19.308 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:47:19.309 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863639 2023-10-21T04:47:19.310 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:19.310 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:47:19.310 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:47:19.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:47:24.111 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:47:24.109+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:47:29.315 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:29.325 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:29.326 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:47:29.327 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:30.632 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:47:30.633 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:47:30.634 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863650 2023-10-21T04:47:30.634 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:30.634 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:47:30.634 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:47:30.635 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:47:40.636 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:47:40.636 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:40.636 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:40.646 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:42.000 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:47:42.000 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:47:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863662 2023-10-21T04:47:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:47:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:47:42.002 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:47:52.005 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:47:52.005 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:52.016 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:52.030 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:53.215 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:47:53.226 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:47:53.227 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863673 2023-10-21T04:47:53.228 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:53.228 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:47:53.228 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:47:53.228 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:48:03.231 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:48:03.231 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:03.232 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:03.245 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:04.299 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:48:04.308 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:48:04.310 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863684 2023-10-21T04:48:04.310 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:04.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:48:04.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:48:04.311 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:48:12.423 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T04:48:12.423 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:48:12.424 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:48:12.424 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:48:12.424 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T04:48:12.424 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:48:12.424 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:48:12.425 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:48:12.425 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:48:12.425 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:48:12.425 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T04:48:12.425 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:48:12.425 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:48:12.426 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:48:12.419+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:48:14.313 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:14.314 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:14.314 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:14.319 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:48:15.475 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:48:15.476 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:48:15.476 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863695 2023-10-21T04:48:15.477 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:15.477 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:48:15.477 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:48:15.477 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:48:25.485 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:25.485 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:25.489 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:48:25.489 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:26.703 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:48:26.704 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:48:26.717 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863706 2023-10-21T04:48:26.717 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:26.717 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:48:26.718 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:48:26.718 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:48:36.719 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:48:36.719 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:36.720 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:36.736 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:37.693 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:48:37.701 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:48:37.707 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863717 2023-10-21T04:48:37.708 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:37.708 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:48:37.708 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:48:37.708 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:48:42.083 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:48:42.074+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:48:47.717 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:48:47.718 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:47.718 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:47.728 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:48.918 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:48:48.918 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:48:48.919 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863728 2023-10-21T04:48:48.919 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:48.919 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:48:48.920 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:48:48.920 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:48:58.927 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:48:58.927 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:58.928 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:58.931 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:00.018 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:49:00.014+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:49:00.024 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:49:00.022+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:49:00.024 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:49:00.022+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:49:00.042 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:49:00.038+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:49:00.042 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:49:00.038+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:49:00.043 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:49:00.038+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:49:00.043 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:49:00.038+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:49:00.043 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:49:00.038+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:49:00.043 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:49:00.038+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:49:00.043 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:49:00.038+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:49:00.044 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:49:00.038+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:49:00.186 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:49:00.186 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:49:00.187 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863740 2023-10-21T04:49:00.187 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:00.187 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:49:00.187 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:49:00.187 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:49:10.195 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:10.196 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:49:10.205 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:10.213 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:11.301 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:49:11.301 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:49:11.301 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863751 2023-10-21T04:49:11.302 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:11.302 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:49:11.302 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:49:11.302 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:49:13.701 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:49:13.697+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:49:21.310 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:21.311 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:49:21.316 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:21.320 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:22.631 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:49:22.652 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:49:22.653 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863762 2023-10-21T04:49:22.654 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:22.654 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:49:22.654 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:49:22.654 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:49:32.656 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:32.657 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:32.660 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:32.666 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:49:33.940 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:49:33.942 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:49:33.943 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863773 2023-10-21T04:49:33.943 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:33.943 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:49:33.944 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:49:33.944 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:49:43.945 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:49:43.952 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:43.953 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:43.964 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:45.107 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:49:45.107 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:49:45.108 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863785 2023-10-21T04:49:45.108 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:45.108 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:49:45.109 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:49:45.109 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:49:55.113 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:49:55.113 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:55.123 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:55.125 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:56.491 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:49:56.492 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:49:56.492 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863796 2023-10-21T04:49:56.492 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:56.492 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:49:56.493 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:49:56.493 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:50:06.496 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:06.499 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:06.505 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:06.507 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:50:07.898 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:50:07.899 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:50:07.900 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863807 2023-10-21T04:50:07.901 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:07.901 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:50:07.901 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:50:07.901 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:50:17.903 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:50:17.908 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:17.911 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:17.924 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:18.926 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:50:18.944 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:50:18.945 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863818 2023-10-21T04:50:18.945 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:18.946 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:50:18.946 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:50:18.946 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:50:25.594 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:50:25.592+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:50:25.595 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:50:25.592+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:50:25.595 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:50:25.592+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:50:28.947 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:50:28.948 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:28.954 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:28.963 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:30.221 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:50:30.235 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:50:30.237 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863830 2023-10-21T04:50:30.237 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:30.238 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:50:30.238 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:50:30.238 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:50:40.244 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:50:40.244 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:40.245 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:40.257 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:41.411 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:50:41.416 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:50:41.417 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863841 2023-10-21T04:50:41.418 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:41.418 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:50:41.418 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:50:41.418 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:50:46.861 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:50:46.851+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:50:46.862 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:50:46.851+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:50:46.862 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:50:46.855+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:50:51.420 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:50:51.423 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:51.435 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:51.436 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:52.502 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:50:52.503 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:50:52.503 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863852 2023-10-21T04:50:52.504 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:52.504 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:50:52.504 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:50:52.504 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:50:55.962 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:50:55.959+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:50:57.968 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:50:57.963+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:51:02.505 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:02.506 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:02.507 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:02.508 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:51:03.724 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:51:03.726 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:51:03.727 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863863 2023-10-21T04:51:03.727 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:03.727 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:51:03.728 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:51:03.728 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:51:13.729 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:51:13.729 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:13.730 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:13.764 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:14.958 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:51:14.962 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:51:14.963 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863874 2023-10-21T04:51:14.963 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:14.963 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:51:14.964 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:51:14.964 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:51:24.965 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:24.966 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:51:24.970 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:24.982 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:26.260 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:51:26.262 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:51:26.273 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863886 2023-10-21T04:51:26.273 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:26.273 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:51:26.274 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:51:26.274 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:51:36.275 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:36.275 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:51:36.279 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:36.284 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:37.704 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:51:37.704 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:51:37.705 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863897 2023-10-21T04:51:37.705 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:37.705 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:51:37.706 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:51:37.706 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:51:47.707 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:47.708 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:47.711 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:47.715 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:51:49.068 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:51:49.069 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:51:49.071 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863909 2023-10-21T04:51:49.071 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:49.071 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:51:49.071 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:51:49.072 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:51:59.073 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:51:59.075 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:59.079 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:59.090 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:00.158 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:52:00.159 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:52:00.160 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863920 2023-10-21T04:52:00.160 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:00.160 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:52:00.160 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:52:00.161 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:52:10.163 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:10.163 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:52:10.163 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:10.168 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:11.451 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:52:11.453 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:52:11.455 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863931 2023-10-21T04:52:11.455 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:11.456 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:52:11.456 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:52:11.456 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:52:21.457 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:21.457 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:52:21.457 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:21.459 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:22.774 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:52:22.774 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:52:22.776 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863942 2023-10-21T04:52:22.776 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:22.776 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:52:22.776 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:52:22.777 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:52:32.778 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:32.787 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:32.787 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:32.788 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:52:34.034 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:52:34.035 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:52:34.035 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863954 2023-10-21T04:52:34.035 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:34.036 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:52:34.036 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:52:34.036 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:52:44.039 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:44.041 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:44.042 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:52:44.045 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:45.137 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:52:45.137 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:52:45.138 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863965 2023-10-21T04:52:45.138 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:45.138 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:52:45.139 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:52:45.139 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:52:55.140 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:55.143 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:52:55.143 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:55.156 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:56.264 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:52:56.264 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:52:56.265 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863976 2023-10-21T04:52:56.265 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:56.265 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:52:56.266 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:52:56.266 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:53:06.268 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:53:06.268 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:06.279 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:06.280 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:07.729 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:53:07.731 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:53:07.734 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863987 2023-10-21T04:53:07.734 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:07.735 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:53:07.735 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:53:07.735 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:53:17.738 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:53:17.738 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:17.738 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:17.754 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:18.710 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:53:18.710 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:53:18.710 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697863998 2023-10-21T04:53:18.711 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:18.711 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:53:18.711 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:53:18.711 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:53:28.715 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:28.715 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:28.718 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:28.721 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:53:29.974 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:53:29.974 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:53:29.982 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864009 2023-10-21T04:53:29.982 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:29.983 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:53:29.983 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:53:29.983 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:53:39.998 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:53:40.001 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:40.002 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:40.005 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:41.252 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:53:41.259 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:53:41.260 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864021 2023-10-21T04:53:41.260 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:41.261 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:53:41.261 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:53:41.261 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:53:51.271 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:53:51.272 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:51.272 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:51.281 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:52.429 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:53:52.430 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:53:52.439 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864032 2023-10-21T04:53:52.440 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:52.440 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:53:52.440 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:53:52.440 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:54:02.449 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:54:02.449 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:02.461 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:02.474 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:03.672 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:54:03.675 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:54:03.676 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864043 2023-10-21T04:54:03.677 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:03.677 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:54:03.677 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:54:03.677 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:54:13.689 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:54:13.697 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:13.697 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:13.716 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:15.155 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:54:15.155 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:54:15.156 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864055 2023-10-21T04:54:15.156 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:15.157 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:54:15.157 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:54:15.157 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:54:16.950 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:54:16.942+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:54:16.950 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:54:16.946+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:54:16.950 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:54:16.946+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:54:16.951 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:54:16.946+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:54:16.951 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:54:16.946+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:54:16.951 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:54:16.946+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:54:16.951 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:54:16.946+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:54:25.158 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:54:25.158 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:25.159 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:26.548 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:54:26.548 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:54:26.548 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864066 2023-10-21T04:54:26.549 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:26.549 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:54:26.549 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:54:26.549 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:54:36.553 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:54:36.553 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:36.554 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:36.554 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:37.622 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:54:37.629 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:54:37.631 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864077 2023-10-21T04:54:37.631 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:37.631 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:54:37.631 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:54:37.631 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:54:47.633 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:54:47.634 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:47.637 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:47.647 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:48.864 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:54:48.864 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:54:48.864 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864088 2023-10-21T04:54:48.864 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:48.865 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:54:48.865 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:54:48.865 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:54:58.871 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:58.873 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:54:58.876 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:58.913 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:00.176 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:55:00.178 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:55:00.179 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864100 2023-10-21T04:55:00.179 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:00.179 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:55:00.179 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:55:00.180 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:55:05.008 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:55:05.005+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:55:10.190 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:10.190 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:10.200 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:55:10.219 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:11.274 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:55:11.275 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:55:11.276 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864111 2023-10-21T04:55:11.277 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:11.277 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:55:11.277 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:55:11.277 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:55:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:55:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:21.281 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:22.556 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:55:22.556 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:55:22.557 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864122 2023-10-21T04:55:22.557 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:22.558 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:55:22.558 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:55:22.558 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:55:32.559 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:32.566 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:55:32.566 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:32.581 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:33.769 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:55:33.778 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:55:33.781 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864133 2023-10-21T04:55:33.782 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:33.782 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:55:33.782 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:55:33.782 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:55:43.804 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:55:43.805 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:43.805 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:43.805 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:45.057 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:55:45.057 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:55:45.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864145 2023-10-21T04:55:45.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:45.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:55:45.059 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:55:45.059 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:55:50.388 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:55:50.383+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:55:55.066 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:55.066 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:55.067 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:55:55.067 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:56.095 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:55:56.109 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:55:56.110 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864156 2023-10-21T04:55:56.110 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:56.110 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:55:56.110 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:55:56.111 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:56:06.135 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:06.139 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:56:06.139 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:06.144 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:07.428 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:56:07.429 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:56:07.429 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864167 2023-10-21T04:56:07.429 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:07.430 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:56:07.430 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:56:07.430 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:56:17.432 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:17.435 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:17.437 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:17.457 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:56:18.777 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:56:18.777 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:56:18.784 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864178 2023-10-21T04:56:18.784 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:18.784 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:56:18.784 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:56:18.785 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:56:28.793 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:56:28.800 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:28.804 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:28.812 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:30.022 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:56:30.023 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:56:30.023 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864190 2023-10-21T04:56:30.023 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:30.024 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:56:30.024 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:56:30.024 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:56:40.025 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:40.026 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:56:40.042 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:40.042 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:41.241 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:56:41.242 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:56:41.244 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864201 2023-10-21T04:56:41.244 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:41.244 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:56:41.245 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:56:41.245 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:56:51.251 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:56:51.252 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:51.252 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:51.256 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:52.572 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:56:52.573 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:56:52.573 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864212 2023-10-21T04:56:52.574 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:52.574 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:56:52.574 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:56:52.574 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:57:02.579 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:57:02.579 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:02.581 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:02.588 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:03.358 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:03.354+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:57:03.358 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:03.354+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:57:03.650 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:57:03.650 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:57:03.651 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864223 2023-10-21T04:57:03.651 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:03.651 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:57:03.652 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:57:03.652 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:57:13.652 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:13.653 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:13.668 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:57:13.669 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:14.881 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:57:14.881 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:57:14.882 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864234 2023-10-21T04:57:14.882 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:14.882 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:57:14.883 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:57:14.883 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:57:21.917 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T04:57:21.913+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T04:57:24.884 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:24.885 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:24.892 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:57:24.897 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:26.057 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:57:26.057 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:57:26.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864246 2023-10-21T04:57:26.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:26.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:57:26.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:57:26.058 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:57:34.699 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:34.693+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T04:57:34.700 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:34.693+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T04:57:34.700 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:34.693+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T04:57:34.700 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:34.693+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:57:34.700 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:34.693+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:57:34.700 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:34.693+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:57:34.701 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:34.693+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T04:57:34.701 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:34.693+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T04:57:34.701 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:34.693+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T04:57:34.701 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:34.693+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T04:57:34.701 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:34.693+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T04:57:34.702 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:34.693+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T04:57:36.060 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:36.060 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:57:36.066 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:36.068 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:37.215 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:57:37.219 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:57:37.229 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864257 2023-10-21T04:57:37.230 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:37.230 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:57:37.230 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:57:37.230 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:57:45.880 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:45.877+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T04:57:45.892 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:45.881+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T04:57:45.892 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T04:57:45.881+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T04:57:47.232 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:47.232 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:57:47.232 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:47.236 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:48.498 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:57:48.498 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:57:48.499 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864268 2023-10-21T04:57:48.499 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:48.499 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:57:48.499 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:57:48.500 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:57:58.501 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:58.504 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:58.504 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:57:58.518 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:59.678 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:57:59.678 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:57:59.679 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864279 2023-10-21T04:57:59.679 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:59.679 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:57:59.679 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:57:59.679 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:58:09.687 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:58:09.688 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:09.688 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:09.696 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:10.937 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:58:10.938 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:58:10.940 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864290 2023-10-21T04:58:10.940 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:10.940 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:58:10.941 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:58:10.941 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:58:20.949 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:20.949 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:20.949 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:20.951 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:58:22.215 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:58:22.221 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:58:22.232 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864302 2023-10-21T04:58:22.232 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:22.232 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:58:22.232 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:58:22.233 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:58:32.237 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:32.238 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:58:32.247 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:32.249 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:33.409 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:58:33.409 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:58:33.411 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864313 2023-10-21T04:58:33.411 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:33.411 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:58:33.411 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:58:33.412 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:58:43.412 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:58:43.413 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:43.415 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:43.423 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:44.753 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:58:44.753 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:58:44.754 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864324 2023-10-21T04:58:44.754 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:44.754 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:58:44.755 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:58:44.755 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:58:54.759 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:54.760 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:54.760 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:54.762 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:58:55.994 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:58:55.994 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:58:55.995 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864335 2023-10-21T04:58:55.995 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:55.995 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:58:55.995 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:58:55.996 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:59:06.001 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:06.002 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:59:06.008 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:06.012 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:07.103 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:59:07.104 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:59:07.104 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864347 2023-10-21T04:59:07.104 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:07.105 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:59:07.105 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:59:07.105 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:59:17.109 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:59:17.109 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:17.110 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:17.112 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:18.191 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:59:18.191 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:59:18.193 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864358 2023-10-21T04:59:18.193 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:18.193 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:59:18.193 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:59:18.193 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:59:28.202 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:59:28.202 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:28.202 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:28.204 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:29.530 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:59:29.531 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:59:29.531 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864369 2023-10-21T04:59:29.532 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:29.532 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:59:29.532 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:59:29.532 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:59:39.540 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:39.544 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:39.547 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:39.548 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:59:40.715 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:59:40.716 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:59:40.717 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864380 2023-10-21T04:59:40.717 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:40.717 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:59:40.718 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:59:40.718 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T04:59:50.719 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:50.724 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:50.724 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:50.732 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T04:59:51.899 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T04:59:51.900 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T04:59:51.900 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864391 2023-10-21T04:59:51.900 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:51.901 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T04:59:51.901 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T04:59:51.901 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:00:01.909 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:00:01.909 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:01.910 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:01.910 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:03.089 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:00:03.090 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:00:03.090 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864403 2023-10-21T05:00:03.090 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:03.091 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:00:03.091 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:00:03.091 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:00:13.104 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:00:13.104 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:13.105 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:13.118 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:14.254 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:00:14.254 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:00:14.255 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864414 2023-10-21T05:00:14.255 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:14.255 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:00:14.255 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:00:14.256 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:00:17.485 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:17.473+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:00:17.487 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:17.473+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:00:17.487 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:17.477+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:00:17.549 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:17.545+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:00:17.549 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:17.545+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:00:23.980 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T05:00:23.973+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T05:00:24.278 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:24.279 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:24.281 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:00:24.285 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:25.403 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:00:25.412 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:00:25.413 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864425 2023-10-21T05:00:25.413 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:25.413 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:00:25.414 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:00:25.414 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:00:35.415 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:35.415 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:00:35.419 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:35.435 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:36.771 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:00:36.772 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:00:36.777 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864436 2023-10-21T05:00:36.777 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:36.778 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:00:36.778 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:00:36.778 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:00:46.791 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:00:46.791 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:46.802 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:46.808 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:48.080 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:00:48.080 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:00:48.081 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864448 2023-10-21T05:00:48.082 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:48.082 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:00:48.082 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:00:48.082 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:00:58.084 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:58.087 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:58.097 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:58.127 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:00:58.372 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T05:00:58.372 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:00:58.373 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T05:00:58.373 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:00:58.373 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T05:00:58.373 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:00:58.374 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:00:58.374 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:00:58.374 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:00:58.374 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T05:00:58.374 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T05:00:58.375 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:00:58.375 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:00:58.368+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:00:59.369 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:00:59.370 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:00:59.371 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864459 2023-10-21T05:00:59.371 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:59.371 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:00:59.371 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:00:59.371 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:01:09.389 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:01:09.389 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:09.390 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:09.390 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:10.515 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:01:10.515 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:01:10.516 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864470 2023-10-21T05:01:10.517 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:10.517 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:01:10.518 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:01:10.518 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:01:20.533 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:20.534 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:20.538 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:20.560 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:01:21.552 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:01:21.555 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:01:21.560 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864481 2023-10-21T05:01:21.560 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:21.560 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:01:21.561 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:01:21.561 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:01:31.562 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:31.563 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:01:31.564 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:31.564 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:32.993 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:01:33.015 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:01:33.016 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864493 2023-10-21T05:01:33.017 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:33.017 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:01:33.017 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:01:33.017 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:01:34.349 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:01:34.347+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:01:34.350 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:01:34.347+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:01:34.350 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:01:34.347+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:01:34.355 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:01:34.351+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:01:34.355 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:01:34.351+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:01:34.355 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:01:34.351+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:01:43.018 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:01:43.019 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:43.021 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:43.028 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:44.253 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:01:44.264 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:01:44.266 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864504 2023-10-21T05:01:44.267 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:44.267 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:01:44.267 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:01:44.267 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:01:54.273 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:54.274 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:54.281 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:01:54.282 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:55.356 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:01:55.357 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:01:55.359 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864515 2023-10-21T05:01:55.360 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:55.360 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:01:55.360 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:01:55.361 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:02:05.363 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:05.363 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:02:05.364 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:05.365 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:06.604 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:02:06.605 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:02:06.605 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864526 2023-10-21T05:02:06.605 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:06.606 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:02:06.606 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:02:06.606 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:02:16.613 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:16.614 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:16.614 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:02:16.617 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:17.929 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:02:17.931 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:02:17.933 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864537 2023-10-21T05:02:17.933 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:17.933 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:02:17.933 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:02:17.934 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:02:27.935 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:27.946 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:27.952 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:02:27.957 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:29.600 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:02:29.601 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:02:29.601 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864549 2023-10-21T05:02:29.601 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:29.602 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:02:29.602 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:02:29.602 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:02:30.249 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T05:02:30.246+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T05:02:39.606 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:02:39.608 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:39.615 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:39.617 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:40.963 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:02:40.963 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:02:40.964 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864560 2023-10-21T05:02:40.964 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:40.964 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:02:40.965 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:02:40.965 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:02:50.966 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:02:50.966 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:50.969 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:50.980 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:52.254 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:02:52.255 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:02:52.255 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864572 2023-10-21T05:02:52.256 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:52.256 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:02:52.256 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:02:52.256 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:03:02.257 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:03:02.257 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:02.267 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:02.267 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:03.556 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:03:03.556 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:03:03.557 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864583 2023-10-21T05:03:03.557 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:03.558 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:03:03.558 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:03:03.558 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:03:13.560 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:03:13.561 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:13.561 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:13.563 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:14.907 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:03:14.908 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:03:14.908 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864594 2023-10-21T05:03:14.909 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:14.909 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:03:14.909 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:03:14.909 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:03:24.910 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:03:24.911 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:24.913 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:24.918 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:26.147 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:03:26.150 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:03:26.152 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864606 2023-10-21T05:03:26.152 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:26.152 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:03:26.152 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:03:26.153 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:03:36.164 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:03:36.165 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:36.166 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:36.166 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:37.396 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:03:37.396 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:03:37.396 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864617 2023-10-21T05:03:37.397 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:37.397 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:03:37.397 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:03:37.397 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:03:47.400 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:47.400 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:03:47.415 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:47.422 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:48.314 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:03:48.314 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:03:48.315 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864628 2023-10-21T05:03:48.315 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:48.315 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:03:48.315 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:03:48.315 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:03:48.441 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T05:03:48.441 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:03:48.441 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T05:03:48.441 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:03:48.442 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T05:03:48.442 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:03:48.442 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:03:48.442 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:03:48.442 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:03:48.443 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T05:03:48.443 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T05:03:48.443 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T05:03:48.443 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:03:48.443 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:03:48.436+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:03:58.343 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:58.344 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:58.345 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:58.345 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:03:59.670 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:03:59.671 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:03:59.671 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864639 2023-10-21T05:03:59.671 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:59.672 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:03:59.672 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:03:59.672 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:04:09.679 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:04:09.681 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:04:09.683 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:04:09.689 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:04:10.980 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:04:10.984 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:04:10.985 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864650 2023-10-21T05:04:10.986 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:04:10.986 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:04:10.986 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:04:10.986 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:04:20.996 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:04:20.997 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:04:21.000 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:04:21.026 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:04:22.162 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:04:22.162 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:04:22.163 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864662 2023-10-21T05:04:22.163 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:04:22.163 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:04:22.164 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:04:22.164 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:04:32.167 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:04:32.180 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:04:32.180 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:04:32.201 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:04:33.496 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:04:33.500 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:04:33.501 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864673 2023-10-21T05:04:33.502 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:04:33.502 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:04:33.502 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:04:33.502 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:04:38.187 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:38.183+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T05:04:38.188 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:38.183+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:04:38.197 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:38.191+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T05:04:38.197 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:38.191+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:04:38.197 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:38.191+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:04:38.198 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:38.191+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:04:38.198 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:38.191+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:04:38.200 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:38.195+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:04:38.201 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:38.195+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T05:04:38.201 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:38.195+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:04:38.201 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:38.195+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:04:43.514 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:04:43.514 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:04:43.515 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:04:43.515 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:04:44.585 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:04:44.588 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:04:44.589 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864684 2023-10-21T05:04:44.589 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:04:44.589 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:04:44.590 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:04:44.590 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:04:53.121 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:53.114+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:04:53.121 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:04:53.114+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:04:54.596 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:04:54.599 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:04:54.599 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:04:54.619 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:04:55.927 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:04:55.928 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:04:55.928 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864695 2023-10-21T05:04:55.928 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:04:55.929 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:04:55.929 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:04:55.929 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:05:05.932 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:05:05.933 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:05:05.938 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:05:05.948 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:05:07.241 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:05:07.244 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:05:07.245 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864707 2023-10-21T05:05:07.245 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:05:07.246 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:05:07.246 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:05:07.246 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:05:17.247 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:05:17.248 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:05:17.248 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:05:17.260 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:05:18.523 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:05:18.532 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:05:18.533 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864718 2023-10-21T05:05:18.534 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:05:18.534 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:05:18.534 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:05:18.534 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:05:28.541 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:05:28.541 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:05:28.541 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:05:28.542 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:05:29.504 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:05:29.505 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:05:29.514 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864729 2023-10-21T05:05:29.515 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:05:29.515 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:05:29.515 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:05:29.515 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:05:39.518 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:05:39.521 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:05:39.522 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:05:39.535 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:05:40.632 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:05:40.640 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:05:40.642 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864740 2023-10-21T05:05:40.642 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:05:40.642 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:05:40.642 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:05:40.643 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:05:41.057 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T05:05:41.053+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T05:05:50.644 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:05:50.644 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:05:50.647 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:05:50.662 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:05:51.848 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:05:51.848 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:05:51.860 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864751 2023-10-21T05:05:51.861 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:05:51.861 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:05:51.861 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:05:51.861 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:06:01.870 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:06:01.873 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:06:01.875 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:06:01.879 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:06:03.062 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:06:03.063 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:06:03.064 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864763 2023-10-21T05:06:03.065 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:06:03.065 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:06:03.065 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:06:03.065 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:06:03.831 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:03.824+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:06:03.841 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:03.836+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:06:13.070 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:06:13.070 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:06:13.070 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:06:13.074 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:06:14.385 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:06:14.385 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:06:14.394 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864774 2023-10-21T05:06:14.395 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:06:14.395 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:06:14.395 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:06:14.395 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:06:24.400 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:06:24.405 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:06:24.410 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:06:24.411 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:06:25.474 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:06:25.474 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:06:25.475 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864785 2023-10-21T05:06:25.475 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:06:25.475 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:06:25.475 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:06:25.475 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:06:35.476 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:06:35.476 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:06:35.485 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:06:35.492 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:06:36.677 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:06:36.678 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:06:36.679 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864796 2023-10-21T05:06:36.679 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:06:36.679 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:06:36.680 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:06:36.680 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:06:45.778 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T05:06:45.778 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:06:45.779 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T05:06:45.779 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:06:45.779 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T05:06:45.779 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:06:45.779 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:06:45.780 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:06:45.780 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:06:45.780 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T05:06:45.780 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T05:06:45.780 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T05:06:45.781 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:06:45.781 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:06:45.775+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:06:46.681 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:06:46.681 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:06:46.688 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:06:46.694 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:06:47.741 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:06:47.753 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:06:47.755 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864807 2023-10-21T05:06:47.755 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:06:47.755 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:06:47.755 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:06:47.756 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:06:57.761 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:06:57.761 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:06:57.763 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:06:57.764 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:06:59.029 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:06:59.030 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:06:59.031 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864819 2023-10-21T05:06:59.031 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:06:59.031 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ -n 172.21.15.172:0/3003149756 ]] 2023-10-21T05:06:59.031 INFO:tasks.workunit.client.0.smithi172.stderr:+ [[ 172.21.15.172:0/3003149756 != \1\7\2\.\2\1\.\1\5\.\1\7\2\:\0\/\3\0\0\3\1\4\9\7\5\6 ]] 2023-10-21T05:06:59.033 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:07:09.032 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:07:09.042 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:07:09.058 INFO:tasks.workunit.client.0.smithi172.stderr:++ jq '.active_clients[]' 2023-10-21T05:07:09.077 INFO:tasks.workunit.client.0.smithi172.stderr:++ ceph mgr dump 2023-10-21T05:07:10.183 INFO:tasks.workunit.client.0.smithi172.stderr:+ CLIENT_ADDR=172.21.15.172:0/3003149756 2023-10-21T05:07:10.183 INFO:tasks.workunit.client.0.smithi172.stderr:++ date +%s 2023-10-21T05:07:10.184 INFO:tasks.workunit.client.0.smithi172.stderr:+ CURRENT_TIME=1697864830 2023-10-21T05:07:10.184 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:07:10.184 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i = 1 )) 2023-10-21T05:07:10.184 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:07:10.184 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:10.454 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:07:10.451+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:10.455 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:10.459 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:07:20.469 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:07:20.470 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:07:20.470 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:20.585 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:07:20.582+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:20.586 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:20.590 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:07:30.596 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:07:30.597 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:07:30.597 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:30.729 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:30.730 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:07:30.722+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:30.735 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:07:38.017 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:07:38.006+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T05:07:38.017 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:07:38.010+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:07:38.020 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:07:38.014+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T05:07:38.023 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:07:38.018+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:07:38.023 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:07:38.018+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:07:38.023 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:07:38.018+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:07:38.043 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:07:38.034+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:07:38.045 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:07:38.034+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T05:07:38.045 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:07:38.034+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T05:07:38.045 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:07:38.034+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:07:38.045 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:07:38.034+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:07:40.743 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:07:40.744 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:07:40.744 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:40.851 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:07:40.846+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:40.852 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:40.857 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:07:50.859 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:07:50.860 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:07:50.860 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:50.915 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:07:50.910+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:50.916 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:50.929 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:08:00.930 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:08:00.931 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:08:00.931 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:08:00.981 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:08:00.977+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:08:00.982 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:08:00.996 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:08:10.997 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:08:10.998 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:08:10.999 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:08:11.075 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:08:11.069+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:08:11.075 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:08:11.084 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:08:21.085 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:08:21.085 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:08:21.086 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:08:21.173 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:08:21.169+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:08:21.173 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:08:21.190 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:08:31.191 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:08:31.192 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:08:31.192 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:08:31.327 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:08:31.321+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:08:31.327 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:08:31.354 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:08:41.355 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:08:41.355 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:08:41.356 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:08:41.466 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:08:41.461+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:08:41.466 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:08:41.477 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:08:51.478 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:08:51.478 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:08:51.478 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:08:51.637 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:08:51.636+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:08:51.637 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:08:51.656 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:09:01.657 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:09:01.658 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:09:01.658 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:09:01.754 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:09:01.752+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:09:01.757 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:09:01.770 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:09:11.043 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T05:09:11.040+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T05:09:11.772 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:09:11.772 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:09:11.772 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:09:11.869 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:09:11.868+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:09:11.870 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:09:11.877 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:09:21.879 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:09:21.879 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:09:21.880 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:09:21.971 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:09:21.968+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:09:21.971 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:09:21.984 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:09:28.982 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T05:09:28.979+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T05:09:31.986 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:09:31.986 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:09:31.987 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:09:32.090 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:09:32.092 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:09:32.087+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:09:32.102 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:09:41.713 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.711+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T05:09:41.714 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.711+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:09:41.727 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.715+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T05:09:41.728 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.715+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:09:41.728 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.723+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T05:09:41.728 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.723+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:09:41.728 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.723+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:09:41.729 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.723+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:09:41.732 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.727+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:09:41.732 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.727+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T05:09:41.732 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.727+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T05:09:41.732 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.727+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T05:09:41.732 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.727+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:09:41.733 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:09:41.727+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:09:42.103 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:09:42.103 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:09:42.103 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:09:42.196 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:09:42.191+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:09:42.196 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:09:42.200 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:09:52.202 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:09:52.202 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:09:52.202 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:09:52.396 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:09:52.391+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:09:52.397 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:09:52.401 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:10:02.402 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:10:02.403 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:10:02.403 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:10:02.536 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:10:02.531+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:10:02.536 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:10:02.549 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:10:12.554 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:10:12.555 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:10:12.555 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:10:12.704 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:10:12.698+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:10:12.704 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:10:12.720 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:10:14.913 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:14.910+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:10:14.913 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:14.910+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:10:14.923 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:14.918+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:10:14.928 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:14.922+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:10:14.928 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:14.922+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:10:14.948 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:14.942+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:10:14.948 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:14.942+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:10:17.908 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T05:10:17.908 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:10:17.909 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T05:10:17.909 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:10:17.909 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T05:10:17.909 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:10:17.909 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:10:17.910 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:10:17.910 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:10:17.910 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T05:10:17.910 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T05:10:17.910 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T05:10:17.911 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:10:17.911 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:10:17.902+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:10:19.254 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T05:10:19.246+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T05:10:22.745 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:10:22.745 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:10:22.745 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:10:22.884 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:10:22.878+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:10:22.885 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:10:22.909 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:10:32.913 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:10:32.913 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:10:32.913 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:10:33.108 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:10:33.102+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:10:33.108 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:10:33.130 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:10:43.133 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:10:43.133 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:10:43.134 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:10:43.276 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:10:43.270+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:10:43.276 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:10:43.291 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:10:53.292 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:10:53.292 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:10:53.293 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:10:53.465 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:10:53.461+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:10:53.465 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:10:53.487 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:10:58.526 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T05:10:58.517+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T05:11:03.496 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:11:03.496 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:11:03.496 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:11:03.605 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:11:03.597+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:11:03.612 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:11:03.618 INFO:tasks.workunit.client.0.smithi172.stderr:+ sleep 10 2023-10-21T05:11:13.620 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i++ )) 2023-10-21T05:11:13.620 INFO:tasks.workunit.client.0.smithi172.stderr:+ (( i <= 24 )) 2023-10-21T05:11:13.620 INFO:tasks.workunit.client.0.smithi172.stderr:+ grep 'every 2m' 2023-10-21T05:11:13.621 INFO:tasks.workunit.client.0.smithi172.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-10-21T05:11:13.718 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-21T05:11:13.713+0000 7fa7c2e5d640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:11:13.718 INFO:tasks.workunit.client.0.smithi172.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:11:13.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T05:11:13.740 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-21T05:11:13.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-21T05:11:14.530 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi172 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-21T05:11:14.600 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=25638d60a1044fe1a8f0842d6f95c1f0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi172 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-21T05:11:14.603 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-21T05:11:14.671 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-21T05:11:14.714 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-21T05:11:14.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:15.345 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:11:15.346 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:11:15.374 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2034,"stamp":"2023-10-21T05:11:13.622485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21351036771,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":871577,"num_read_kb":3367811,"num_write":856429,"num_write_kb":3426207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61983,"ondisk_log_size":61983,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":33769856,"kb_used_data":31291560,"kb_used_omap":36,"kb_used_meta":2478235,"kb_avail":249345664,"statfs":{"total":289910292480,"available":255329959936,"internally_reserved":0,"allocated":32042557440,"data_stored":32041470116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":2537713100},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,21,17,10,1,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":98304,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2741,"num_read_kb":10840,"num_write":2592,"num_write_kb":10368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002089"},"pg_stats":[{"pgid":"2.7","version":"54'4315","reported_seq":6947,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:12.768268+0000","last_change":"2023-10-21T04:05:16.931966+0000","last_active":"2023-10-21T05:11:12.768268+0000","last_peered":"2023-10-21T05:11:12.768268+0000","last_clean":"2023-10-21T05:11:12.768268+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:11:12.768268+0000","last_undegraded":"2023-10-21T05:11:12.768268+0000","last_fullsized":"2023-10-21T05:11:12.768268+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:51:01.446979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7156","reported_seq":11704,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:12.788396+0000","last_change":"2023-10-21T04:05:16.933396+0000","last_active":"2023-10-21T05:11:12.788396+0000","last_peered":"2023-10-21T05:11:12.788396+0000","last_clean":"2023-10-21T05:11:12.788396+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:11:12.788396+0000","last_undegraded":"2023-10-21T05:11:12.788396+0000","last_fullsized":"2023-10-21T05:11:12.788396+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":7156,"log_dups_size":0,"ondisk_log_size":7156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:03:36.143248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1459","reported_seq":2408,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:10.693492+0000","last_change":"2023-10-21T04:05:16.931967+0000","last_active":"2023-10-21T05:11:10.693492+0000","last_peered":"2023-10-21T05:11:10.693492+0000","last_clean":"2023-10-21T05:11:10.693492+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:11:10.693492+0000","last_undegraded":"2023-10-21T05:11:10.693492+0000","last_fullsized":"2023-10-21T05:11:10.693492+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:34:31.728606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1471","reported_seq":2541,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:10.696719+0000","last_change":"2023-10-21T04:05:16.933170+0000","last_active":"2023-10-21T05:11:10.696719+0000","last_peered":"2023-10-21T05:11:10.696719+0000","last_clean":"2023-10-21T05:11:10.696719+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:11:10.696719+0000","last_undegraded":"2023-10-21T05:11:10.696719+0000","last_fullsized":"2023-10-21T05:11:10.696719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":1471,"log_dups_size":0,"ondisk_log_size":1471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:05:47.346028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'222056","reported_seq":445868,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:10.488009+0000","last_change":"2023-10-21T04:05:28.011414+0000","last_active":"2023-10-21T05:11:10.488009+0000","last_peered":"2023-10-21T05:11:10.488009+0000","last_clean":"2023-10-21T05:11:10.488009+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:11:10.488009+0000","last_undegraded":"2023-10-21T05:11:10.488009+0000","last_fullsized":"2023-10-21T05:11:10.488009+0000","mapping_epoch":17,"log_start":"51'211992","ondisk_log_start":"51'211992","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:54:40.049012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5520306176,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":223708,"num_read_kb":868832,"num_write":222056,"num_write_kb":888224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1544","reported_seq":2502,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:03.698877+0000","last_change":"2023-10-21T04:05:16.934828+0000","last_active":"2023-10-21T05:11:03.698877+0000","last_peered":"2023-10-21T05:11:03.698877+0000","last_clean":"2023-10-21T05:11:03.698877+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:11:03.698877+0000","last_undegraded":"2023-10-21T05:11:03.698877+0000","last_fullsized":"2023-10-21T05:11:03.698877+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":1544,"log_dups_size":0,"ondisk_log_size":1544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:53:30.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'207418","reported_seq":417643,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:13.462374+0000","last_change":"2023-10-21T04:05:28.000682+0000","last_active":"2023-10-21T05:11:13.462374+0000","last_peered":"2023-10-21T05:11:13.462374+0000","last_clean":"2023-10-21T05:11:13.462374+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:11:13.462374+0000","last_undegraded":"2023-10-21T05:11:13.462374+0000","last_fullsized":"2023-10-21T05:11:13.462374+0000","mapping_epoch":17,"log_start":"51'197353","ondisk_log_start":"51'197353","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:36:45.628195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5188358144,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":210119,"num_read_kb":815668,"num_write":207418,"num_write_kb":829672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"50'19","reported_seq":953,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:03.593556+0000","last_change":"2023-10-21T04:05:16.933866+0000","last_active":"2023-10-21T05:11:03.593556+0000","last_peered":"2023-10-21T05:11:03.593556+0000","last_clean":"2023-10-21T05:11:03.593556+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:11:03.593556+0000","last_undegraded":"2023-10-21T05:11:03.593556+0000","last_fullsized":"2023-10-21T05:11:03.593556+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:27:23.376634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'215598","reported_seq":433771,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:10.467808+0000","last_change":"2023-10-21T04:05:28.011228+0000","last_active":"2023-10-21T05:11:10.467808+0000","last_peered":"2023-10-21T05:11:10.467808+0000","last_clean":"2023-10-21T05:11:10.467808+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:11:10.467808+0000","last_undegraded":"2023-10-21T05:11:10.467808+0000","last_fullsized":"2023-10-21T05:11:10.467808+0000","mapping_epoch":17,"log_start":"51'205574","ondisk_log_start":"51'205574","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:27:08.469770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5382967296,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":218068,"num_read_kb":847476,"num_write":215598,"num_write_kb":862392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"50'59","reported_seq":3546,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:08.718993+0000","last_change":"2023-10-21T04:05:16.932868+0000","last_active":"2023-10-21T05:11:08.718993+0000","last_peered":"2023-10-21T05:11:08.718993+0000","last_clean":"2023-10-21T05:11:08.718993+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:11:08.718993+0000","last_undegraded":"2023-10-21T05:11:08.718993+0000","last_fullsized":"2023-10-21T05:11:08.718993+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:13:31.077070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'210957","reported_seq":424977,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:13.455351+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:11:13.455351+0000","last_peered":"2023-10-21T05:11:13.455351+0000","last_clean":"2023-10-21T05:11:13.455351+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:11:13.455351+0000","last_undegraded":"2023-10-21T05:11:13.455351+0000","last_fullsized":"2023-10-21T05:11:13.455351+0000","mapping_epoch":17,"log_start":"51'200946","ondisk_log_start":"51'200946","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:23:31.433537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5258813459,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":213916,"num_read_kb":830928,"num_write":210957,"num_write_kb":843822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5699","reported_seq":9150,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:12.801417+0000","last_change":"2023-10-21T04:05:16.931948+0000","last_active":"2023-10-21T05:11:12.801417+0000","last_peered":"2023-10-21T05:11:12.801417+0000","last_clean":"2023-10-21T05:11:12.801417+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:11:12.801417+0000","last_undegraded":"2023-10-21T05:11:12.801417+0000","last_fullsized":"2023-10-21T05:11:12.801417+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":5699,"log_dups_size":0,"ondisk_log_size":5699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:33:13.546258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":212,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-21T05:09:57.653109+0000","last_change":"2023-10-21T04:05:12.086073+0000","last_active":"2023-10-21T05:09:57.653109+0000","last_peered":"2023-10-21T05:09:57.653109+0000","last_clean":"2023-10-21T05:09:57.653109+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:09:57.653109+0000","last_undegraded":"2023-10-21T05:09:57.653109+0000","last_fullsized":"2023-10-21T05:09:57.653109+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:10.868962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:10.868962+0000","last_clean_scrub_stamp":"2023-10-21T04:05:10.868962+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:25:27.169589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21350445075,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":865811,"num_read_kb":3362904,"num_write":856029,"num_write_kb":3424110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32039636992,"data_stored":32039636992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40164,"ondisk_log_size":40164,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":21722,"ondisk_log_size":21722,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739161,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11255488,"kb_used_data":10430072,"kb_used_omap":8,"kb_used_meta":825399,"kb_avail":83116352,"statfs":{"total":96636764160,"available":85111144448,"internally_reserved":0,"allocated":10680393728,"data_stored":10680089197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":845209245},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:31 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.98799999999999999,"15min":0.91900000000000004},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.14699999999999999},"max":{"1min":5.7169999999999996,"5min":22.969000000000001,"15min":22.969000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.84099999999999997,"15min":0.67300000000000004},"min":{"1min":0.20999999999999999,"5min":0.17899999999999999,"15min":0.16500000000000001},"max":{"1min":3.3330000000000002,"5min":22.835000000000001,"15min":22.835000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42399999999999999,"15min":0.55300000000000005},"min":{"1min":0.19,"5min":0.186,"15min":0.184},"max":{"1min":0.56499999999999995,"5min":4.5309999999999997,"15min":7.8399999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.94399999999999995,"15min":0.81699999999999995},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19600000000000001},"max":{"1min":5.2140000000000004,"5min":6.5410000000000004,"15min":8.5150000000000006},"last":5.2290000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739160,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11254268,"kb_used_data":10430772,"kb_used_omap":15,"kb_used_meta":823472,"kb_avail":83117572,"statfs":{"total":96636764160,"available":85112393728,"internally_reserved":0,"allocated":10681110528,"data_stored":10680692732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":843235718},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,6,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.84499999999999997,"15min":0.80000000000000004},"min":{"1min":0.14999999999999999,"5min":0.13,"15min":0.13},"max":{"1min":3.7730000000000001,"5min":10.627000000000001,"15min":11.225},"last":0.157},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.185,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.8479999999999999,"5min":5.3209999999999997,"15min":7.1639999999999997},"last":0.28199999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.89900000000000002,"15min":0.80000000000000004},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":4.8920000000000003,"5min":6.6180000000000003,"15min":8.5389999999999997},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.88800000000000001,"15min":0.82299999999999995},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.673,"5min":9.1950000000000003,"15min":9.1950000000000003},"last":0.39800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739161,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11260100,"kb_used_data":10430716,"kb_used_omap":12,"kb_used_meta":829363,"kb_avail":83111740,"statfs":{"total":96636764160,"available":85106421760,"internally_reserved":0,"allocated":10681053184,"data_stored":10680688187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":849268137},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,14,9,5,1,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.60399999999999998,"15min":0.57099999999999995},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":6.5759999999999996,"5min":6.5780000000000003,"15min":11.773},"last":0.626},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69099999999999995,"15min":0.63200000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.94799999999999995}]},{"osd":2,"last update":"Sat Oct 21 05:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.70699999999999996,"15min":0.65600000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":6.7409999999999997,"5min":6.7409999999999997,"15min":6.7409999999999997},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.343,"5min":0.88800000000000001,"15min":0.79400000000000004},"min":{"1min":0.21199999999999999,"5min":0.20000000000000001,"15min":0.19700000000000001},"max":{"1min":7.8710000000000004,"5min":8.8510000000000009,"15min":10.074999999999999},"last":0.17599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10679873536,"data_stored":10679873536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10679881728,"data_stored":10679881728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10679881728,"data_stored":10679881728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:15.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:11:16.033 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:11:16.084 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2035,"stamp":"2023-10-21T05:11:15.622877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21351049059,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":872068,"num_read_kb":3369755,"num_write":856948,"num_write_kb":3428283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61999,"ondisk_log_size":61999,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":33774004,"kb_used_data":31291548,"kb_used_omap":36,"kb_used_meta":2482395,"kb_avail":249341516,"statfs":{"total":289910292480,"available":255325712384,"internally_reserved":0,"allocated":32042545152,"data_stored":32041457828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":2541972940},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,1,13,12,14,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":98304,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2709,"num_read_kb":10716,"num_write":2599,"num_write_kb":10396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002123"},"pg_stats":[{"pgid":"2.7","version":"54'4315","reported_seq":6947,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:12.768268+0000","last_change":"2023-10-21T04:05:16.931966+0000","last_active":"2023-10-21T05:11:12.768268+0000","last_peered":"2023-10-21T05:11:12.768268+0000","last_clean":"2023-10-21T05:11:12.768268+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:11:12.768268+0000","last_undegraded":"2023-10-21T05:11:12.768268+0000","last_fullsized":"2023-10-21T05:11:12.768268+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:51:01.446979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7156","reported_seq":11704,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:12.788396+0000","last_change":"2023-10-21T04:05:16.933396+0000","last_active":"2023-10-21T05:11:12.788396+0000","last_peered":"2023-10-21T05:11:12.788396+0000","last_clean":"2023-10-21T05:11:12.788396+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:11:12.788396+0000","last_undegraded":"2023-10-21T05:11:12.788396+0000","last_fullsized":"2023-10-21T05:11:12.788396+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":7156,"log_dups_size":0,"ondisk_log_size":7156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:03:36.143248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1459","reported_seq":2408,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:10.693492+0000","last_change":"2023-10-21T04:05:16.931967+0000","last_active":"2023-10-21T05:11:10.693492+0000","last_peered":"2023-10-21T05:11:10.693492+0000","last_clean":"2023-10-21T05:11:10.693492+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:11:10.693492+0000","last_undegraded":"2023-10-21T05:11:10.693492+0000","last_fullsized":"2023-10-21T05:11:10.693492+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:34:31.728606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1471","reported_seq":2541,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:10.696719+0000","last_change":"2023-10-21T04:05:16.933170+0000","last_active":"2023-10-21T05:11:10.696719+0000","last_peered":"2023-10-21T05:11:10.696719+0000","last_clean":"2023-10-21T05:11:10.696719+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:11:10.696719+0000","last_undegraded":"2023-10-21T05:11:10.696719+0000","last_fullsized":"2023-10-21T05:11:10.696719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":1471,"log_dups_size":0,"ondisk_log_size":1471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:05:47.346028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'222309","reported_seq":446378,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:15.561664+0000","last_change":"2023-10-21T04:05:28.011414+0000","last_active":"2023-10-21T05:11:15.561664+0000","last_peered":"2023-10-21T05:11:15.561664+0000","last_clean":"2023-10-21T05:11:15.561664+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:11:15.561664+0000","last_undegraded":"2023-10-21T05:11:15.561664+0000","last_fullsized":"2023-10-21T05:11:15.561664+0000","mapping_epoch":17,"log_start":"51'212294","ondisk_log_start":"51'212294","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:54:40.049012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5520318464,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":223967,"num_read_kb":869856,"num_write":222309,"num_write_kb":889236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1547","reported_seq":2506,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:12.866303+0000","last_change":"2023-10-21T04:05:16.934828+0000","last_active":"2023-10-21T05:11:12.866303+0000","last_peered":"2023-10-21T05:11:12.866303+0000","last_clean":"2023-10-21T05:11:12.866303+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:11:12.866303+0000","last_undegraded":"2023-10-21T05:11:12.866303+0000","last_fullsized":"2023-10-21T05:11:12.866303+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":1547,"log_dups_size":0,"ondisk_log_size":1547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:53:30.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'207418","reported_seq":417643,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:13.462374+0000","last_change":"2023-10-21T04:05:28.000682+0000","last_active":"2023-10-21T05:11:13.462374+0000","last_peered":"2023-10-21T05:11:13.462374+0000","last_clean":"2023-10-21T05:11:13.462374+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:11:13.462374+0000","last_undegraded":"2023-10-21T05:11:13.462374+0000","last_fullsized":"2023-10-21T05:11:13.462374+0000","mapping_epoch":17,"log_start":"51'197353","ondisk_log_start":"51'197353","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:36:45.628195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5188358144,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":210119,"num_read_kb":815668,"num_write":207418,"num_write_kb":829672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"50'19","reported_seq":953,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:03.593556+0000","last_change":"2023-10-21T04:05:16.933866+0000","last_active":"2023-10-21T05:11:03.593556+0000","last_peered":"2023-10-21T05:11:03.593556+0000","last_clean":"2023-10-21T05:11:03.593556+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:11:03.593556+0000","last_undegraded":"2023-10-21T05:11:03.593556+0000","last_fullsized":"2023-10-21T05:11:03.593556+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:27:23.376634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'215864","reported_seq":434263,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:15.549607+0000","last_change":"2023-10-21T04:05:28.011228+0000","last_active":"2023-10-21T05:11:15.549607+0000","last_peered":"2023-10-21T05:11:15.549607+0000","last_clean":"2023-10-21T05:11:15.549607+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:11:15.549607+0000","last_undegraded":"2023-10-21T05:11:15.549607+0000","last_fullsized":"2023-10-21T05:11:15.549607+0000","mapping_epoch":17,"log_start":"51'205778","ondisk_log_start":"51'205778","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:27:08.469770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5382967296,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":218300,"num_read_kb":848396,"num_write":215864,"num_write_kb":863456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"50'59","reported_seq":3546,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:08.718993+0000","last_change":"2023-10-21T04:05:16.932868+0000","last_active":"2023-10-21T05:11:08.718993+0000","last_peered":"2023-10-21T05:11:08.718993+0000","last_clean":"2023-10-21T05:11:08.718993+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:11:08.718993+0000","last_undegraded":"2023-10-21T05:11:08.718993+0000","last_fullsized":"2023-10-21T05:11:08.718993+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:13:31.077070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'210957","reported_seq":424977,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:13.455351+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:11:13.455351+0000","last_peered":"2023-10-21T05:11:13.455351+0000","last_clean":"2023-10-21T05:11:13.455351+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:11:13.455351+0000","last_undegraded":"2023-10-21T05:11:13.455351+0000","last_fullsized":"2023-10-21T05:11:13.455351+0000","mapping_epoch":17,"log_start":"51'200946","ondisk_log_start":"51'200946","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:23:31.433537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5258813459,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":213916,"num_read_kb":830928,"num_write":210957,"num_write_kb":843822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5699","reported_seq":9150,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:12.801417+0000","last_change":"2023-10-21T04:05:16.931948+0000","last_active":"2023-10-21T05:11:12.801417+0000","last_peered":"2023-10-21T05:11:12.801417+0000","last_clean":"2023-10-21T05:11:12.801417+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:11:12.801417+0000","last_undegraded":"2023-10-21T05:11:12.801417+0000","last_fullsized":"2023-10-21T05:11:12.801417+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":5699,"log_dups_size":0,"ondisk_log_size":5699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:33:13.546258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":212,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-21T05:09:57.653109+0000","last_change":"2023-10-21T04:05:12.086073+0000","last_active":"2023-10-21T05:09:57.653109+0000","last_peered":"2023-10-21T05:09:57.653109+0000","last_clean":"2023-10-21T05:09:57.653109+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:09:57.653109+0000","last_undegraded":"2023-10-21T05:09:57.653109+0000","last_fullsized":"2023-10-21T05:09:57.653109+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:10.868962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:10.868962+0000","last_clean_scrub_stamp":"2023-10-21T04:05:10.868962+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:25:27.169589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21350457363,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":866302,"num_read_kb":3364848,"num_write":856548,"num_write_kb":3426186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32039686144,"data_stored":32039686144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40177,"ondisk_log_size":40177,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":21725,"ondisk_log_size":21725,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739161,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11255488,"kb_used_data":10430072,"kb_used_omap":8,"kb_used_meta":825399,"kb_avail":83116352,"statfs":{"total":96636764160,"available":85111144448,"internally_reserved":0,"allocated":10680393728,"data_stored":10680089197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":845209245},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:31 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.98799999999999999,"15min":0.91900000000000004},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.14699999999999999},"max":{"1min":5.7169999999999996,"5min":22.969000000000001,"15min":22.969000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.84099999999999997,"15min":0.67300000000000004},"min":{"1min":0.20999999999999999,"5min":0.17899999999999999,"15min":0.16500000000000001},"max":{"1min":3.3330000000000002,"5min":22.835000000000001,"15min":22.835000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42399999999999999,"15min":0.55300000000000005},"min":{"1min":0.19,"5min":0.186,"15min":0.184},"max":{"1min":0.56499999999999995,"5min":4.5309999999999997,"15min":7.8399999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.94399999999999995,"15min":0.81699999999999995},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19600000000000001},"max":{"1min":5.2140000000000004,"5min":6.5410000000000004,"15min":8.5150000000000006},"last":5.2290000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739160,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11254268,"kb_used_data":10430772,"kb_used_omap":15,"kb_used_meta":823472,"kb_avail":83117572,"statfs":{"total":96636764160,"available":85112393728,"internally_reserved":0,"allocated":10681110528,"data_stored":10680692732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":843235718},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,6,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.84499999999999997,"15min":0.80000000000000004},"min":{"1min":0.14999999999999999,"5min":0.13,"15min":0.13},"max":{"1min":3.7730000000000001,"5min":10.627000000000001,"15min":11.225},"last":0.157},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.185,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.8479999999999999,"5min":5.3209999999999997,"15min":7.1639999999999997},"last":0.28199999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.89900000000000002,"15min":0.80000000000000004},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":4.8920000000000003,"5min":6.6180000000000003,"15min":8.5389999999999997},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.88800000000000001,"15min":0.82299999999999995},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.673,"5min":9.1950000000000003,"15min":9.1950000000000003},"last":0.39800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739162,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11264248,"kb_used_data":10430704,"kb_used_omap":12,"kb_used_meta":833523,"kb_avail":83107592,"statfs":{"total":96636764160,"available":85102174208,"internally_reserved":0,"allocated":10681040896,"data_stored":10680675899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":853527977},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,0,6,4,9,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.60399999999999998,"15min":0.57099999999999995},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":6.5759999999999996,"5min":6.5780000000000003,"15min":11.773},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69099999999999995,"15min":0.63200000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.70699999999999996,"15min":0.65600000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":6.7409999999999997,"5min":6.7409999999999997,"15min":6.7409999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.343,"5min":0.88800000000000001,"15min":0.79400000000000004},"min":{"1min":0.21199999999999999,"5min":0.20000000000000001,"15min":0.19700000000000001},"max":{"1min":7.8710000000000004,"5min":8.8510000000000009,"15min":10.074999999999999},"last":0.309}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10679922688,"data_stored":10679922688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10679881728,"data_stored":10679881728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10679881728,"data_stored":10679881728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:16.086 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-21T05:11:16.086 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:16.671 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:11:16.672 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:11:16.701 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2035,"stamp":"2023-10-21T05:11:15.622877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21351049059,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":872068,"num_read_kb":3369755,"num_write":856948,"num_write_kb":3428283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61999,"ondisk_log_size":61999,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":33774004,"kb_used_data":31291548,"kb_used_omap":36,"kb_used_meta":2482395,"kb_avail":249341516,"statfs":{"total":289910292480,"available":255325712384,"internally_reserved":0,"allocated":32042545152,"data_stored":32041457828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":2541972940},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,1,13,12,14,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":98304,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2709,"num_read_kb":10716,"num_write":2599,"num_write_kb":10396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002123"},"pg_stats":[{"pgid":"2.7","version":"54'4315","reported_seq":6947,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:12.768268+0000","last_change":"2023-10-21T04:05:16.931966+0000","last_active":"2023-10-21T05:11:12.768268+0000","last_peered":"2023-10-21T05:11:12.768268+0000","last_clean":"2023-10-21T05:11:12.768268+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:11:12.768268+0000","last_undegraded":"2023-10-21T05:11:12.768268+0000","last_fullsized":"2023-10-21T05:11:12.768268+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:51:01.446979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7156","reported_seq":11704,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:12.788396+0000","last_change":"2023-10-21T04:05:16.933396+0000","last_active":"2023-10-21T05:11:12.788396+0000","last_peered":"2023-10-21T05:11:12.788396+0000","last_clean":"2023-10-21T05:11:12.788396+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:11:12.788396+0000","last_undegraded":"2023-10-21T05:11:12.788396+0000","last_fullsized":"2023-10-21T05:11:12.788396+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":7156,"log_dups_size":0,"ondisk_log_size":7156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:03:36.143248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1459","reported_seq":2408,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:10.693492+0000","last_change":"2023-10-21T04:05:16.931967+0000","last_active":"2023-10-21T05:11:10.693492+0000","last_peered":"2023-10-21T05:11:10.693492+0000","last_clean":"2023-10-21T05:11:10.693492+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:11:10.693492+0000","last_undegraded":"2023-10-21T05:11:10.693492+0000","last_fullsized":"2023-10-21T05:11:10.693492+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:34:31.728606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1471","reported_seq":2541,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:10.696719+0000","last_change":"2023-10-21T04:05:16.933170+0000","last_active":"2023-10-21T05:11:10.696719+0000","last_peered":"2023-10-21T05:11:10.696719+0000","last_clean":"2023-10-21T05:11:10.696719+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:11:10.696719+0000","last_undegraded":"2023-10-21T05:11:10.696719+0000","last_fullsized":"2023-10-21T05:11:10.696719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":1471,"log_dups_size":0,"ondisk_log_size":1471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:05:47.346028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'222309","reported_seq":446378,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:15.561664+0000","last_change":"2023-10-21T04:05:28.011414+0000","last_active":"2023-10-21T05:11:15.561664+0000","last_peered":"2023-10-21T05:11:15.561664+0000","last_clean":"2023-10-21T05:11:15.561664+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:11:15.561664+0000","last_undegraded":"2023-10-21T05:11:15.561664+0000","last_fullsized":"2023-10-21T05:11:15.561664+0000","mapping_epoch":17,"log_start":"51'212294","ondisk_log_start":"51'212294","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:54:40.049012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5520318464,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":223967,"num_read_kb":869856,"num_write":222309,"num_write_kb":889236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1547","reported_seq":2506,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:12.866303+0000","last_change":"2023-10-21T04:05:16.934828+0000","last_active":"2023-10-21T05:11:12.866303+0000","last_peered":"2023-10-21T05:11:12.866303+0000","last_clean":"2023-10-21T05:11:12.866303+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:11:12.866303+0000","last_undegraded":"2023-10-21T05:11:12.866303+0000","last_fullsized":"2023-10-21T05:11:12.866303+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":1547,"log_dups_size":0,"ondisk_log_size":1547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:53:30.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'207418","reported_seq":417643,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:13.462374+0000","last_change":"2023-10-21T04:05:28.000682+0000","last_active":"2023-10-21T05:11:13.462374+0000","last_peered":"2023-10-21T05:11:13.462374+0000","last_clean":"2023-10-21T05:11:13.462374+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:11:13.462374+0000","last_undegraded":"2023-10-21T05:11:13.462374+0000","last_fullsized":"2023-10-21T05:11:13.462374+0000","mapping_epoch":17,"log_start":"51'197353","ondisk_log_start":"51'197353","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:36:45.628195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5188358144,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":210119,"num_read_kb":815668,"num_write":207418,"num_write_kb":829672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"50'19","reported_seq":953,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:03.593556+0000","last_change":"2023-10-21T04:05:16.933866+0000","last_active":"2023-10-21T05:11:03.593556+0000","last_peered":"2023-10-21T05:11:03.593556+0000","last_clean":"2023-10-21T05:11:03.593556+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:11:03.593556+0000","last_undegraded":"2023-10-21T05:11:03.593556+0000","last_fullsized":"2023-10-21T05:11:03.593556+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:27:23.376634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'215864","reported_seq":434263,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:15.549607+0000","last_change":"2023-10-21T04:05:28.011228+0000","last_active":"2023-10-21T05:11:15.549607+0000","last_peered":"2023-10-21T05:11:15.549607+0000","last_clean":"2023-10-21T05:11:15.549607+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:11:15.549607+0000","last_undegraded":"2023-10-21T05:11:15.549607+0000","last_fullsized":"2023-10-21T05:11:15.549607+0000","mapping_epoch":17,"log_start":"51'205778","ondisk_log_start":"51'205778","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:27:08.469770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5382967296,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":218300,"num_read_kb":848396,"num_write":215864,"num_write_kb":863456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"50'59","reported_seq":3546,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:08.718993+0000","last_change":"2023-10-21T04:05:16.932868+0000","last_active":"2023-10-21T05:11:08.718993+0000","last_peered":"2023-10-21T05:11:08.718993+0000","last_clean":"2023-10-21T05:11:08.718993+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:11:08.718993+0000","last_undegraded":"2023-10-21T05:11:08.718993+0000","last_fullsized":"2023-10-21T05:11:08.718993+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:13:31.077070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'210957","reported_seq":424977,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:13.455351+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:11:13.455351+0000","last_peered":"2023-10-21T05:11:13.455351+0000","last_clean":"2023-10-21T05:11:13.455351+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:11:13.455351+0000","last_undegraded":"2023-10-21T05:11:13.455351+0000","last_fullsized":"2023-10-21T05:11:13.455351+0000","mapping_epoch":17,"log_start":"51'200946","ondisk_log_start":"51'200946","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:23:31.433537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5258813459,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":213916,"num_read_kb":830928,"num_write":210957,"num_write_kb":843822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5699","reported_seq":9150,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:12.801417+0000","last_change":"2023-10-21T04:05:16.931948+0000","last_active":"2023-10-21T05:11:12.801417+0000","last_peered":"2023-10-21T05:11:12.801417+0000","last_clean":"2023-10-21T05:11:12.801417+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:11:12.801417+0000","last_undegraded":"2023-10-21T05:11:12.801417+0000","last_fullsized":"2023-10-21T05:11:12.801417+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":5699,"log_dups_size":0,"ondisk_log_size":5699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:33:13.546258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":212,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-21T05:09:57.653109+0000","last_change":"2023-10-21T04:05:12.086073+0000","last_active":"2023-10-21T05:09:57.653109+0000","last_peered":"2023-10-21T05:09:57.653109+0000","last_clean":"2023-10-21T05:09:57.653109+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:09:57.653109+0000","last_undegraded":"2023-10-21T05:09:57.653109+0000","last_fullsized":"2023-10-21T05:09:57.653109+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:10.868962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:10.868962+0000","last_clean_scrub_stamp":"2023-10-21T04:05:10.868962+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:25:27.169589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21350457363,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":866302,"num_read_kb":3364848,"num_write":856548,"num_write_kb":3426186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32039686144,"data_stored":32039686144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40177,"ondisk_log_size":40177,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":21725,"ondisk_log_size":21725,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739161,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11255488,"kb_used_data":10430072,"kb_used_omap":8,"kb_used_meta":825399,"kb_avail":83116352,"statfs":{"total":96636764160,"available":85111144448,"internally_reserved":0,"allocated":10680393728,"data_stored":10680089197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":845209245},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:31 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.98799999999999999,"15min":0.91900000000000004},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.14699999999999999},"max":{"1min":5.7169999999999996,"5min":22.969000000000001,"15min":22.969000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.84099999999999997,"15min":0.67300000000000004},"min":{"1min":0.20999999999999999,"5min":0.17899999999999999,"15min":0.16500000000000001},"max":{"1min":3.3330000000000002,"5min":22.835000000000001,"15min":22.835000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42399999999999999,"15min":0.55300000000000005},"min":{"1min":0.19,"5min":0.186,"15min":0.184},"max":{"1min":0.56499999999999995,"5min":4.5309999999999997,"15min":7.8399999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.94399999999999995,"15min":0.81699999999999995},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19600000000000001},"max":{"1min":5.2140000000000004,"5min":6.5410000000000004,"15min":8.5150000000000006},"last":5.2290000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739160,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11254268,"kb_used_data":10430772,"kb_used_omap":15,"kb_used_meta":823472,"kb_avail":83117572,"statfs":{"total":96636764160,"available":85112393728,"internally_reserved":0,"allocated":10681110528,"data_stored":10680692732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":843235718},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,6,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.84499999999999997,"15min":0.80000000000000004},"min":{"1min":0.14999999999999999,"5min":0.13,"15min":0.13},"max":{"1min":3.7730000000000001,"5min":10.627000000000001,"15min":11.225},"last":0.157},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.185,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.8479999999999999,"5min":5.3209999999999997,"15min":7.1639999999999997},"last":0.28199999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.89900000000000002,"15min":0.80000000000000004},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":4.8920000000000003,"5min":6.6180000000000003,"15min":8.5389999999999997},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.88800000000000001,"15min":0.82299999999999995},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.673,"5min":9.1950000000000003,"15min":9.1950000000000003},"last":0.39800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739162,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11264248,"kb_used_data":10430704,"kb_used_omap":12,"kb_used_meta":833523,"kb_avail":83107592,"statfs":{"total":96636764160,"available":85102174208,"internally_reserved":0,"allocated":10681040896,"data_stored":10680675899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":853527977},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,0,6,4,9,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.60399999999999998,"15min":0.57099999999999995},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":6.5759999999999996,"5min":6.5780000000000003,"15min":11.773},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69099999999999995,"15min":0.63200000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.70699999999999996,"15min":0.65600000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":6.7409999999999997,"5min":6.7409999999999997,"15min":6.7409999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.343,"5min":0.88800000000000001,"15min":0.79400000000000004},"min":{"1min":0.21199999999999999,"5min":0.20000000000000001,"15min":0.19700000000000001},"max":{"1min":7.8710000000000004,"5min":8.8510000000000009,"15min":10.074999999999999},"last":0.309}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10679922688,"data_stored":10679922688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10679881728,"data_stored":10679881728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10679881728,"data_stored":10679881728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:16.703 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T05:11:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:11:17.413 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":54,"fsid":"e51e76a3-8801-445f-ab5b-9b66dbc6ff69","created":"2023-10-21T04:04:59.896377+0000","modified":"2023-10-21T05:10:36.286159+0000","last_up_change":"2023-10-21T04:05:09.780590+0000","last_in_change":"2023-10-21T04:05:00.775907+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T04:05:10.624097+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-21T04:05:15.189413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-10-21T04:05:23.000079+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":42,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cc8c080d-06c7-4b59-abca-a7dff121a69a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6817","nonce":3641467676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6819","nonce":3641467676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6823","nonce":3641467676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":3641467676},{"type":"v1","addr":"172.21.15.172:6821","nonce":3641467676}]},"public_addr":"172.21.15.172:6817/3641467676","cluster_addr":"172.21.15.172:6819/3641467676","heartbeat_back_addr":"172.21.15.172:6823/3641467676","heartbeat_front_addr":"172.21.15.172:6821/3641467676","state":["exists","up"]},{"osd":1,"uuid":"1616f68c-54bc-414c-b3c8-07d7c4e24807","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6801","nonce":1983746680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6803","nonce":1983746680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6807","nonce":1983746680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":1983746680},{"type":"v1","addr":"172.21.15.172:6805","nonce":1983746680}]},"public_addr":"172.21.15.172:6801/1983746680","cluster_addr":"172.21.15.172:6803/1983746680","heartbeat_back_addr":"172.21.15.172:6807/1983746680","heartbeat_front_addr":"172.21.15.172:6805/1983746680","state":["exists","up"]},{"osd":2,"uuid":"e6a20f19-f003-4d2c-8574-778059175260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6809","nonce":920067689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6811","nonce":920067689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6815","nonce":920067689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":920067689},{"type":"v1","addr":"172.21.15.172:6813","nonce":920067689}]},"public_addr":"172.21.15.172:6809/920067689","cluster_addr":"172.21.15.172:6811/920067689","heartbeat_back_addr":"172.21.15.172:6815/920067689","heartbeat_front_addr":"172.21.15.172:6813/920067689","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:05:06.035423+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:05:06.181353+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:05:06.119226+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T05:11:18.458 INFO:tasks.ceph:Scrubbing osd.0 2023-10-21T05:11:18.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-21T05:11:18.830 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-21T05:11:18.831 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-21T05:11:18.831 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-21T05:11:18.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-21T05:11:19.644 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 0 to deep-scrub 2023-10-21T05:11:19.689 INFO:tasks.ceph:Scrubbing osd.1 2023-10-21T05:11:19.690 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-21T05:11:19.993 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-21T05:11:19.993 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-21T05:11:19.993 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-21T05:11:20.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-21T05:11:20.762 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 1 to deep-scrub 2023-10-21T05:11:20.804 INFO:tasks.ceph:Scrubbing osd.2 2023-10-21T05:11:20.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-21T05:11:21.155 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-21T05:11:21.156 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-21T05:11:21.156 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-21T05:11:21.194 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-21T05:11:22.001 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 2 to deep-scrub 2023-10-21T05:11:22.086 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:22.789 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:11:22.790 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:11:22.856 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2038,"stamp":"2023-10-21T05:11:21.625726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21351077731,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":873000,"num_read_kb":3373431,"num_write":857938,"num_write_kb":3432243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62014,"ondisk_log_size":62014,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":33804584,"kb_used_data":31291592,"kb_used_omap":36,"kb_used_meta":2512923,"kb_avail":249310936,"statfs":{"total":289910292480,"available":255294398464,"internally_reserved":0,"allocated":32042590208,"data_stored":32041502884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":2573233612},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,10,4,7,17,14,15,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":69632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2565,"num_read_kb":10148,"num_write":2566,"num_write_kb":10264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003926"},"pg_stats":[{"pgid":"2.7","version":"54'4321","reported_seq":6956,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:14.853913+0000","last_change":"2023-10-21T04:05:16.931966+0000","last_active":"2023-10-21T05:11:14.853913+0000","last_peered":"2023-10-21T05:11:14.853913+0000","last_clean":"2023-10-21T05:11:14.853913+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:11:14.853913+0000","last_undegraded":"2023-10-21T05:11:14.853913+0000","last_fullsized":"2023-10-21T05:11:14.853913+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:51:01.446979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7166","reported_seq":11719,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:14.879424+0000","last_change":"2023-10-21T04:05:16.933396+0000","last_active":"2023-10-21T05:11:14.879424+0000","last_peered":"2023-10-21T05:11:14.879424+0000","last_clean":"2023-10-21T05:11:14.879424+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:11:14.879424+0000","last_undegraded":"2023-10-21T05:11:14.879424+0000","last_fullsized":"2023-10-21T05:11:14.879424+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":7166,"log_dups_size":0,"ondisk_log_size":7166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:03:36.143248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1461","reported_seq":2411,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:14.030112+0000","last_change":"2023-10-21T04:05:16.931967+0000","last_active":"2023-10-21T05:11:14.030112+0000","last_peered":"2023-10-21T05:11:14.030112+0000","last_clean":"2023-10-21T05:11:14.030112+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:11:14.030112+0000","last_undegraded":"2023-10-21T05:11:14.030112+0000","last_fullsized":"2023-10-21T05:11:14.030112+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":1461,"log_dups_size":0,"ondisk_log_size":1461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:34:31.728606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1473","reported_seq":2544,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:14.030020+0000","last_change":"2023-10-21T04:05:16.933170+0000","last_active":"2023-10-21T05:11:14.030020+0000","last_peered":"2023-10-21T05:11:14.030020+0000","last_clean":"2023-10-21T05:11:14.030020+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:11:14.030020+0000","last_undegraded":"2023-10-21T05:11:14.030020+0000","last_fullsized":"2023-10-21T05:11:14.030020+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:05:47.346028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'222573","reported_seq":446873,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:20.604186+0000","last_change":"2023-10-21T04:05:28.011414+0000","last_active":"2023-10-21T05:11:20.604186+0000","last_peered":"2023-10-21T05:11:20.604186+0000","last_clean":"2023-10-21T05:11:20.604186+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:11:20.604186+0000","last_undegraded":"2023-10-21T05:11:20.604186+0000","last_fullsized":"2023-10-21T05:11:20.604186+0000","mapping_epoch":17,"log_start":"51'212495","ondisk_log_start":"51'212495","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5520322560,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":224200,"num_read_kb":870768,"num_write":222573,"num_write_kb":890292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1548","reported_seq":2509,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:19.644957+0000","last_change":"2023-10-21T04:05:16.934828+0000","last_active":"2023-10-21T05:11:19.644957+0000","last_peered":"2023-10-21T05:11:19.644957+0000","last_clean":"2023-10-21T05:11:19.644957+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:11:19.644957+0000","last_undegraded":"2023-10-21T05:11:19.644957+0000","last_fullsized":"2023-10-21T05:11:19.644957+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'207663","reported_seq":418146,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:18.587837+0000","last_change":"2023-10-21T04:05:28.000682+0000","last_active":"2023-10-21T05:11:18.587837+0000","last_peered":"2023-10-21T05:11:18.587837+0000","last_clean":"2023-10-21T05:11:18.587837+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:11:18.587837+0000","last_undegraded":"2023-10-21T05:11:18.587837+0000","last_fullsized":"2023-10-21T05:11:18.587837+0000","mapping_epoch":17,"log_start":"51'197653","ondisk_log_start":"51'197653","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:36:45.628195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5188366336,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":210379,"num_read_kb":816692,"num_write":207663,"num_write_kb":830652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"50'19","reported_seq":954,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:08.717683+0000","last_change":"2023-10-21T04:05:16.933866+0000","last_active":"2023-10-21T05:11:08.717683+0000","last_peered":"2023-10-21T05:11:08.717683+0000","last_clean":"2023-10-21T05:11:08.717683+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:11:08.717683+0000","last_undegraded":"2023-10-21T05:11:08.717683+0000","last_fullsized":"2023-10-21T05:11:08.717683+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:27:23.376634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'216126","reported_seq":434762,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:20.559131+0000","last_change":"2023-10-21T04:05:28.011228+0000","last_active":"2023-10-21T05:11:20.559131+0000","last_peered":"2023-10-21T05:11:20.559131+0000","last_clean":"2023-10-21T05:11:20.559131+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:11:20.559131+0000","last_undegraded":"2023-10-21T05:11:20.559131+0000","last_fullsized":"2023-10-21T05:11:20.559131+0000","mapping_epoch":17,"log_start":"51'206080","ondisk_log_start":"51'206080","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5382967296,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":218529,"num_read_kb":849300,"num_write":216126,"num_write_kb":864504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"50'59","reported_seq":3550,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:13.845060+0000","last_change":"2023-10-21T04:05:16.932868+0000","last_active":"2023-10-21T05:11:13.845060+0000","last_peered":"2023-10-21T05:11:13.845060+0000","last_clean":"2023-10-21T05:11:13.845060+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:11:13.845060+0000","last_undegraded":"2023-10-21T05:11:13.845060+0000","last_fullsized":"2023-10-21T05:11:13.845060+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:13:31.077070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'211176","reported_seq":425406,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:18.600870+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:11:18.600870+0000","last_peered":"2023-10-21T05:11:18.600870+0000","last_clean":"2023-10-21T05:11:18.600870+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:11:18.600870+0000","last_undegraded":"2023-10-21T05:11:18.600870+0000","last_fullsized":"2023-10-21T05:11:18.600870+0000","mapping_epoch":17,"log_start":"51'201147","ondisk_log_start":"51'201147","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:23:31.433537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5258829843,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":214126,"num_read_kb":831764,"num_write":211176,"num_write_kb":844698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5707","reported_seq":9162,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:14.884701+0000","last_change":"2023-10-21T04:05:16.931948+0000","last_active":"2023-10-21T05:11:14.884701+0000","last_peered":"2023-10-21T05:11:14.884701+0000","last_clean":"2023-10-21T05:11:14.884701+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:11:14.884701+0000","last_undegraded":"2023-10-21T05:11:14.884701+0000","last_fullsized":"2023-10-21T05:11:14.884701+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:15.903627+0000","last_clean_scrub_stamp":"2023-10-21T04:05:15.903627+0000","objects_scrubbed":0,"log_size":5707,"log_dups_size":0,"ondisk_log_size":5707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:33:13.546258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":212,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-21T05:09:57.653109+0000","last_change":"2023-10-21T04:05:12.086073+0000","last_active":"2023-10-21T05:09:57.653109+0000","last_peered":"2023-10-21T05:09:57.653109+0000","last_clean":"2023-10-21T05:09:57.653109+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:09:57.653109+0000","last_undegraded":"2023-10-21T05:09:57.653109+0000","last_fullsized":"2023-10-21T05:09:57.653109+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:10.868962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:10.868962+0000","last_clean_scrub_stamp":"2023-10-21T04:05:10.868962+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:25:27.169589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21350486035,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":867234,"num_read_kb":3368524,"num_write":857538,"num_write_kb":3430146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32039739392,"data_stored":32039739392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40163,"ondisk_log_size":40163,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":21754,"ondisk_log_size":21754,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739162,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11259672,"kb_used_data":10430088,"kb_used_omap":8,"kb_used_meta":829559,"kb_avail":83112168,"statfs":{"total":96636764160,"available":85106860032,"internally_reserved":0,"allocated":10680410112,"data_stored":10680105581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":849469085},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,1,5,9,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:31 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.98799999999999999,"15min":0.91900000000000004},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.14699999999999999},"max":{"1min":5.7169999999999996,"5min":22.969000000000001,"15min":22.969000000000001},"last":3.7160000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.84099999999999997,"15min":0.67300000000000004},"min":{"1min":0.20999999999999999,"5min":0.17899999999999999,"15min":0.16500000000000001},"max":{"1min":3.3330000000000002,"5min":22.835000000000001,"15min":22.835000000000001},"last":0.216}]},{"osd":1,"last update":"Sat Oct 21 05:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42399999999999999,"15min":0.55300000000000005},"min":{"1min":0.19,"5min":0.186,"15min":0.184},"max":{"1min":0.56499999999999995,"5min":4.5309999999999997,"15min":7.8399999999999999},"last":3.7669999999999999},{"interface":"front","average":{"1min":0.878,"5min":0.94399999999999995,"15min":0.81699999999999995},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19600000000000001},"max":{"1min":5.2140000000000004,"5min":6.5410000000000004,"15min":8.5150000000000006},"last":0.251}]}]},{"osd":1,"up_from":8,"seq":34359739161,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11276460,"kb_used_data":10430756,"kb_used_omap":15,"kb_used_meta":845680,"kb_avail":83095380,"statfs":{"total":96636764160,"available":85089669120,"internally_reserved":0,"allocated":10681094144,"data_stored":10680676348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":865976710},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,2,1,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.753,"15min":0.79000000000000004},"min":{"1min":0.157,"5min":0.13,"15min":0.13},"max":{"1min":2.3010000000000002,"5min":10.627000000000001,"15min":11.225},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.313,"5min":0.46500000000000002,"15min":0.47999999999999998},"min":{"1min":0.20300000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.55000000000000004,"5min":5.3209999999999997,"15min":7.1639999999999997},"last":0.40400000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.80800000000000005,"15min":0.78100000000000003},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":3.4900000000000002,"5min":6.6180000000000003,"15min":8.5389999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.75900000000000001,"15min":0.80100000000000005},"min":{"1min":0.22600000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.5259999999999998,"5min":6.1669999999999998,"15min":9.1950000000000003},"last":0.47099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739163,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11268452,"kb_used_data":10430748,"kb_used_omap":12,"kb_used_meta":837683,"kb_avail":83103388,"statfs":{"total":96636764160,"available":85097869312,"internally_reserved":0,"allocated":10681085952,"data_stored":10680720955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":857787817},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,6,8,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.60399999999999998,"15min":0.57099999999999995},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":6.5759999999999996,"5min":6.5780000000000003,"15min":11.773},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69099999999999995,"15min":0.63200000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.36299999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.70699999999999996,"15min":0.65600000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":6.7409999999999997,"5min":6.7409999999999997,"15min":6.7409999999999997},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.343,"5min":0.88800000000000001,"15min":0.79400000000000004},"min":{"1min":0.21199999999999999,"5min":0.20000000000000001,"15min":0.19700000000000001},"max":{"1min":7.8710000000000004,"5min":8.8510000000000009,"15min":10.074999999999999},"last":5.5149999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10679906304,"data_stored":10679906304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10679918592,"data_stored":10679918592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10679914496,"data_stored":10679914496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:22.860 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-21T04:05:15.903627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.860 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-21T04:05:15.903627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.861 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-21T04:05:15.903627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.861 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-21T04:05:15.903627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.861 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.862 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-21T04:05:15.903627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.862 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.863 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-21T04:05:15.903627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.863 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.863 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-21T04:05:15.903627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.864 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.864 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-21T04:05:15.903627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.864 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-21T04:05:10.868962+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=10, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:22.865 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:11:42.866 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:43.546 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:11:43.547 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:11:43.579 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2048,"stamp":"2023-10-21T05:11:41.632156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21351188323,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":876762,"num_read_kb":3388351,"num_write":861764,"num_write_kb":3447547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62112,"ondisk_log_size":62112,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":33872736,"kb_used_data":31291720,"kb_used_omap":36,"kb_used_meta":2580955,"kb_avail":249242784,"statfs":{"total":289910292480,"available":255224610816,"internally_reserved":0,"allocated":32042721280,"data_stored":32041633956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":2642898380},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,6,14,10,18,20,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":65536,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2307,"num_read_kb":9140,"num_write":2400,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4139,"num_omap_keys":160,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002003"},"pg_stats":[{"pgid":"2.7","version":"54'4339","reported_seq":6995,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:37.292259+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:11:37.292259+0000","last_peered":"2023-10-21T05:11:37.292259+0000","last_clean":"2023-10-21T05:11:37.292259+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:11:37.292259+0000","last_undegraded":"2023-10-21T05:11:37.292259+0000","last_fullsized":"2023-10-21T05:11:37.292259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7196","reported_seq":11778,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:37.304505+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:11:37.304505+0000","last_peered":"2023-10-21T05:11:37.304505+0000","last_clean":"2023-10-21T05:11:37.304505+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:11:37.304505+0000","last_undegraded":"2023-10-21T05:11:37.304505+0000","last_fullsized":"2023-10-21T05:11:37.304505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7196,"log_dups_size":0,"ondisk_log_size":7196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1470","reported_seq":2433,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:36.994010+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:11:36.994010+0000","last_peered":"2023-10-21T05:11:36.994010+0000","last_clean":"2023-10-21T05:11:36.994010+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:11:36.994010+0000","last_undegraded":"2023-10-21T05:11:36.994010+0000","last_fullsized":"2023-10-21T05:11:36.994010+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1470,"log_dups_size":0,"ondisk_log_size":1470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1489","reported_seq":2573,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:36.998086+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:11:36.998086+0000","last_peered":"2023-10-21T05:11:36.998086+0000","last_clean":"2023-10-21T05:11:36.998086+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:11:36.998086+0000","last_undegraded":"2023-10-21T05:11:36.998086+0000","last_fullsized":"2023-10-21T05:11:36.998086+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1489,"log_dups_size":0,"ondisk_log_size":1489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'223545","reported_seq":448841,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:40.778385+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:11:40.778385+0000","last_peered":"2023-10-21T05:11:40.778385+0000","last_clean":"2023-10-21T05:11:40.778385+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:11:40.778385+0000","last_undegraded":"2023-10-21T05:11:40.778385+0000","last_fullsized":"2023-10-21T05:11:40.778385+0000","mapping_epoch":17,"log_start":"52'213526","ondisk_log_start":"52'213526","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":37,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 16s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5520351232,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":225184,"num_read_kb":874668,"num_write":223545,"num_write_kb":894180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1565","reported_seq":2538,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:37.479765+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:11:37.479765+0000","last_peered":"2023-10-21T05:11:37.479765+0000","last_clean":"2023-10-21T05:11:37.479765+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:11:37.479765+0000","last_undegraded":"2023-10-21T05:11:37.479765+0000","last_fullsized":"2023-10-21T05:11:37.479765+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1565,"log_dups_size":0,"ondisk_log_size":1565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'208623","reported_seq":420030,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:38.939599+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:11:38.939599+0000","last_peered":"2023-10-21T05:11:38.939599+0000","last_clean":"2023-10-21T05:11:38.939599+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:11:38.939599+0000","last_undegraded":"2023-10-21T05:11:38.939599+0000","last_fullsized":"2023-10-21T05:11:38.939599+0000","mapping_epoch":17,"log_start":"51'198564","ondisk_log_start":"51'198564","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 4s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5188395008,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":211300,"num_read_kb":820340,"num_write":208623,"num_write_kb":834492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":972,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:29.203087+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:11:29.203087+0000","last_peered":"2023-10-21T05:11:29.203087+0000","last_clean":"2023-10-21T05:11:29.203087+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:11:29.203087+0000","last_undegraded":"2023-10-21T05:11:29.203087+0000","last_fullsized":"2023-10-21T05:11:29.203087+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'217095","reported_seq":436662,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:40.775522+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:11:40.775522+0000","last_peered":"2023-10-21T05:11:40.775522+0000","last_clean":"2023-10-21T05:11:40.775522+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:11:40.775522+0000","last_undegraded":"2023-10-21T05:11:40.775522+0000","last_fullsized":"2023-10-21T05:11:40.775522+0000","mapping_epoch":17,"log_start":"51'207093","ondisk_log_start":"51'207093","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":26,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 15s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5382991872,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":219452,"num_read_kb":852952,"num_write":217095,"num_write_kb":868380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3582,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:34.314522+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:11:34.314522+0000","last_peered":"2023-10-21T05:11:34.314522+0000","last_clean":"2023-10-21T05:11:34.314522+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:11:34.314522+0000","last_undegraded":"2023-10-21T05:11:34.314522+0000","last_fullsized":"2023-10-21T05:11:34.314522+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'212101","reported_seq":427267,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:38.995882+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:11:38.995882+0000","last_peered":"2023-10-21T05:11:38.995882+0000","last_clean":"2023-10-21T05:11:38.995882+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:11:38.995882+0000","last_undegraded":"2023-10-21T05:11:38.995882+0000","last_fullsized":"2023-10-21T05:11:38.995882+0000","mapping_epoch":17,"log_start":"51'202050","ondisk_log_start":"51'202050","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5258858515,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":215060,"num_read_kb":835484,"num_write":212101,"num_write_kb":848398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5733","reported_seq":9213,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:37.355763+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:11:37.355763+0000","last_peered":"2023-10-21T05:11:37.355763+0000","last_clean":"2023-10-21T05:11:37.355763+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:11:37.355763+0000","last_undegraded":"2023-10-21T05:11:37.355763+0000","last_fullsized":"2023-10-21T05:11:37.355763+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":5733,"log_dups_size":0,"ondisk_log_size":5733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":223,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:22.695089+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:22.695089+0000","last_peered":"2023-10-21T05:11:22.695089+0000","last_clean":"2023-10-21T05:11:22.695089+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:22.695089+0000","last_undegraded":"2023-10-21T05:11:22.695089+0000","last_fullsized":"2023-10-21T05:11:22.695089+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21350596627,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":870996,"num_read_kb":3383444,"num_write":861364,"num_write_kb":3445450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32039882752,"data_stored":32039882752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40131,"ondisk_log_size":40131,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":21882,"ondisk_log_size":21882,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739166,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11277756,"kb_used_data":10430132,"kb_used_omap":8,"kb_used_meta":847607,"kb_avail":83094084,"statfs":{"total":96636764160,"available":85088342016,"internally_reserved":0,"allocated":10680455168,"data_stored":10680150637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":867950237},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.98999999999999999,"15min":0.92800000000000005},"min":{"1min":0.18099999999999999,"5min":0.17499999999999999,"15min":0.14699999999999999},"max":{"1min":3.7160000000000002,"5min":22.969000000000001,"15min":22.969000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.85199999999999998,"15min":0.69599999999999995},"min":{"1min":0.20699999999999999,"5min":0.20200000000000001,"15min":0.16500000000000001},"max":{"1min":4.391,"5min":22.835000000000001,"15min":22.835000000000001},"last":0.375}]},{"osd":1,"last update":"Sat Oct 21 05:11:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.51300000000000001,"15min":0.56399999999999995},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.184},"max":{"1min":3.7669999999999999,"5min":4.5309999999999997,"15min":7.8399999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.0780000000000001,"15min":0.86599999999999999},"min":{"1min":0.251,"5min":0.19700000000000001,"15min":0.19600000000000001},"max":{"1min":5.2290000000000001,"5min":6.5410000000000004,"15min":8.5150000000000006},"last":0.622}]}]},{"osd":1,"up_from":8,"seq":34359739165,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11294564,"kb_used_data":10430812,"kb_used_omap":15,"kb_used_meta":863728,"kb_avail":83077276,"statfs":{"total":96636764160,"available":85071130624,"internally_reserved":0,"allocated":10681151488,"data_stored":10680733692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":884457862},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,10,4,7,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.753,"15min":0.79000000000000004},"min":{"1min":0.157,"5min":0.13,"15min":0.13},"max":{"1min":2.3010000000000002,"5min":10.627000000000001,"15min":11.225},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.313,"5min":0.46500000000000002,"15min":0.47999999999999998},"min":{"1min":0.20300000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.55000000000000004,"5min":5.3209999999999997,"15min":7.1639999999999997},"last":0.23499999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.80800000000000005,"15min":0.78100000000000003},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":3.4900000000000002,"5min":6.6180000000000003,"15min":8.5389999999999997},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.75900000000000001,"15min":0.80100000000000005},"min":{"1min":0.22600000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.5259999999999998,"5min":6.1669999999999998,"15min":9.1950000000000003},"last":0.32400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739167,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11300416,"kb_used_data":10430776,"kb_used_omap":12,"kb_used_meta":869619,"kb_avail":83071424,"statfs":{"total":96636764160,"available":85065138176,"internally_reserved":0,"allocated":10681114624,"data_stored":10680749627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":890490281},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,2,4,10,12,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:11:25 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.57999999999999996,"15min":0.54500000000000004},"min":{"1min":0.216,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":0.626,"5min":6.5780000000000003,"15min":11.773},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.67500000000000004,"15min":0.60599999999999998},"min":{"1min":0.24399999999999999,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":0.94799999999999995,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:11:25 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.77100000000000002,"15min":0.63600000000000001},"min":{"1min":0.25600000000000001,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":3.863,"5min":6.7409999999999997,"15min":6.7409999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.3280000000000001,"5min":0.99099999999999999,"15min":0.84599999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":5.625,"5min":7.8710000000000004,"15min":10.074999999999999},"last":0.314}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10679955456,"data_stored":10679955456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10679963648,"data_stored":10679963648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10679963648,"data_stored":10679963648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:43.581 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:43.581 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:43.581 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:43.582 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:43.582 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:12:03.583 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:12:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:12:04.255 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:12:04.301 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2059,"stamp":"2023-10-21T05:12:03.636311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21351409507,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":880409,"num_read_kb":3402799,"num_write":865444,"num_write_kb":3462267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62244,"ondisk_log_size":62244,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":33963384,"kb_used_data":31292124,"kb_used_omap":36,"kb_used_meta":2671195,"kb_avail":249152136,"statfs":{"total":289910292480,"available":255131787264,"internally_reserved":0,"allocated":32043134976,"data_stored":32042047652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":2735304140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,21,12,17,15,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":163840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1806,"num_read_kb":7140,"num_write":1853,"num_write_kb":7412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002415"},"pg_stats":[{"pgid":"2.7","version":"54'4360","reported_seq":7028,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:55.519202+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:11:55.519202+0000","last_peered":"2023-10-21T05:11:55.519202+0000","last_clean":"2023-10-21T05:11:55.519202+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:11:55.519202+0000","last_undegraded":"2023-10-21T05:11:55.519202+0000","last_fullsized":"2023-10-21T05:11:55.519202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4360,"log_dups_size":0,"ondisk_log_size":4360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7231","reported_seq":11833,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:55.537213+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:11:55.537213+0000","last_peered":"2023-10-21T05:11:55.537213+0000","last_clean":"2023-10-21T05:11:55.537213+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:11:55.537213+0000","last_undegraded":"2023-10-21T05:11:55.537213+0000","last_fullsized":"2023-10-21T05:11:55.537213+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7231,"log_dups_size":0,"ondisk_log_size":7231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1477","reported_seq":2444,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:54.797441+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:11:54.797441+0000","last_peered":"2023-10-21T05:11:54.797441+0000","last_clean":"2023-10-21T05:11:54.797441+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:11:54.797441+0000","last_undegraded":"2023-10-21T05:11:54.797441+0000","last_fullsized":"2023-10-21T05:11:54.797441+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1477,"log_dups_size":0,"ondisk_log_size":1477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1496","reported_seq":2584,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:54.797672+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:11:54.797672+0000","last_peered":"2023-10-21T05:11:54.797672+0000","last_clean":"2023-10-21T05:11:54.797672+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:11:54.797672+0000","last_undegraded":"2023-10-21T05:11:54.797672+0000","last_fullsized":"2023-10-21T05:11:54.797672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1496,"log_dups_size":0,"ondisk_log_size":1496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'224503","reported_seq":450771,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:01.084744+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:12:01.084744+0000","last_peered":"2023-10-21T05:12:01.084744+0000","last_clean":"2023-10-21T05:12:01.084744+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:12:01.084744+0000","last_undegraded":"2023-10-21T05:12:01.084744+0000","last_fullsized":"2023-10-21T05:12:01.084744+0000","mapping_epoch":17,"log_start":"52'214440","ondisk_log_start":"52'214440","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":109,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 37s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5520367616,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":226146,"num_read_kb":878472,"num_write":224503,"num_write_kb":898012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1570","reported_seq":2547,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:59.913641+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:11:59.913641+0000","last_peered":"2023-10-21T05:11:59.913641+0000","last_clean":"2023-10-21T05:11:59.913641+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:11:59.913641+0000","last_undegraded":"2023-10-21T05:11:59.913641+0000","last_fullsized":"2023-10-21T05:11:59.913641+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'209522","reported_seq":421804,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:59.204174+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:11:59.204174+0000","last_peered":"2023-10-21T05:11:59.204174+0000","last_clean":"2023-10-21T05:11:59.204174+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:11:59.204174+0000","last_undegraded":"2023-10-21T05:11:59.204174+0000","last_fullsized":"2023-10-21T05:11:59.204174+0000","mapping_epoch":17,"log_start":"52'199480","ondisk_log_start":"52'199480","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":73,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 24s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5188493312,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":212161,"num_read_kb":823760,"num_write":209522,"num_write_kb":838088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":977,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:54.797819+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:11:54.797819+0000","last_peered":"2023-10-21T05:11:54.797819+0000","last_clean":"2023-10-21T05:11:54.797819+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:11:54.797819+0000","last_undegraded":"2023-10-21T05:11:54.797819+0000","last_fullsized":"2023-10-21T05:11:54.797819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'218022","reported_seq":438498,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:01.077860+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:12:01.077860+0000","last_peered":"2023-10-21T05:12:01.077860+0000","last_clean":"2023-10-21T05:12:01.077860+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:12:01.077860+0000","last_undegraded":"2023-10-21T05:12:01.077860+0000","last_fullsized":"2023-10-21T05:12:01.077860+0000","mapping_epoch":17,"log_start":"52'208008","ondisk_log_start":"52'208008","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":74,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 36s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5383041024,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":220351,"num_read_kb":856512,"num_write":218022,"num_write_kb":872088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3603,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:59.934313+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:11:59.934313+0000","last_peered":"2023-10-21T05:11:59.934313+0000","last_clean":"2023-10-21T05:11:59.934313+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:11:59.934313+0000","last_undegraded":"2023-10-21T05:11:59.934313+0000","last_fullsized":"2023-10-21T05:11:59.934313+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'212997","reported_seq":429088,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:59.211571+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:11:59.211571+0000","last_peered":"2023-10-21T05:11:59.211571+0000","last_clean":"2023-10-21T05:11:59.211571+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:11:59.211571+0000","last_undegraded":"2023-10-21T05:11:59.211571+0000","last_fullsized":"2023-10-21T05:11:59.211571+0000","mapping_epoch":17,"log_start":"52'202956","ondisk_log_start":"52'202956","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5258915859,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":215985,"num_read_kb":839148,"num_write":212997,"num_write_kb":851982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5761","reported_seq":9257,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:55.547868+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:11:55.547868+0000","last_peered":"2023-10-21T05:11:55.547868+0000","last_clean":"2023-10-21T05:11:55.547868+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:11:55.547868+0000","last_undegraded":"2023-10-21T05:11:55.547868+0000","last_fullsized":"2023-10-21T05:11:55.547868+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":5761,"log_dups_size":0,"ondisk_log_size":5761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":224,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:22.695206+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:22.695206+0000","last_peered":"2023-10-21T05:11:22.695206+0000","last_clean":"2023-10-21T05:11:22.695206+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:22.695206+0000","last_undegraded":"2023-10-21T05:11:22.695206+0000","last_fullsized":"2023-10-21T05:11:22.695206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21350817811,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":874643,"num_read_kb":3397892,"num_write":865044,"num_write_kb":3460170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32040267776,"data_stored":32040267776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40160,"ondisk_log_size":40160,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":21985,"ondisk_log_size":21985,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739171,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11313984,"kb_used_data":10430260,"kb_used_omap":8,"kb_used_meta":883703,"kb_avail":83057856,"statfs":{"total":96636764160,"available":85051244544,"internally_reserved":0,"allocated":10680586240,"data_stored":10680281709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":904912541},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.98999999999999999,"15min":0.92800000000000005},"min":{"1min":0.18099999999999999,"5min":0.17499999999999999,"15min":0.14699999999999999},"max":{"1min":3.7160000000000002,"5min":22.969000000000001,"15min":22.969000000000001},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.85199999999999998,"15min":0.69599999999999995},"min":{"1min":0.20699999999999999,"5min":0.20200000000000001,"15min":0.16500000000000001},"max":{"1min":4.391,"5min":22.835000000000001,"15min":22.835000000000001},"last":0.378}]},{"osd":1,"last update":"Sat Oct 21 05:11:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.51300000000000001,"15min":0.56399999999999995},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.184},"max":{"1min":3.7669999999999999,"5min":4.5309999999999997,"15min":7.8399999999999999},"last":0.20899999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.0780000000000001,"15min":0.86599999999999999},"min":{"1min":0.251,"5min":0.19700000000000001,"15min":0.19600000000000001},"max":{"1min":5.2290000000000001,"5min":6.5410000000000004,"15min":8.5150000000000006},"last":0.314}]}]},{"osd":1,"up_from":8,"seq":34359739169,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11330832,"kb_used_data":10430984,"kb_used_omap":15,"kb_used_meta":899824,"kb_avail":83041008,"statfs":{"total":96636764160,"available":85033992192,"internally_reserved":0,"allocated":10681327616,"data_stored":10680909820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":921420166},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,7,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.753,"15min":0.79000000000000004},"min":{"1min":0.157,"5min":0.13,"15min":0.13},"max":{"1min":2.3010000000000002,"5min":10.627000000000001,"15min":11.225},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.313,"5min":0.46500000000000002,"15min":0.47999999999999998},"min":{"1min":0.20300000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.55000000000000004,"5min":5.3209999999999997,"15min":7.1639999999999997},"last":0.25800000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.80800000000000005,"15min":0.78100000000000003},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":3.4900000000000002,"5min":6.6180000000000003,"15min":8.5389999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.75900000000000001,"15min":0.80100000000000005},"min":{"1min":0.22600000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.5259999999999998,"5min":6.1669999999999998,"15min":9.1950000000000003},"last":0.35699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739171,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11318568,"kb_used_data":10430880,"kb_used_omap":12,"kb_used_meta":887667,"kb_avail":83053272,"statfs":{"total":96636764160,"available":85046550528,"internally_reserved":0,"allocated":10681221120,"data_stored":10680856123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":908971433},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,11,3,11,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:11:25 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.57999999999999996,"15min":0.54500000000000004},"min":{"1min":0.216,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":0.626,"5min":6.5780000000000003,"15min":11.773},"last":0.31},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.67500000000000004,"15min":0.60599999999999998},"min":{"1min":0.24399999999999999,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":0.94799999999999995,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:11:25 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.77100000000000002,"15min":0.63600000000000001},"min":{"1min":0.25600000000000001,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":3.863,"5min":6.7409999999999997,"15min":6.7409999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.3280000000000001,"5min":0.99099999999999999,"15min":0.84599999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":5.625,"5min":7.8710000000000004,"15min":10.074999999999999},"last":0.22900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10680086528,"data_stored":10680086528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10680041472,"data_stored":10680041472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10680139776,"data_stored":10680139776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:12:04.303 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:04.304 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:04.304 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:04.305 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:04.305 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:12:06.504 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:06.496+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:12:06.506 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:06.504+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:12:06.506 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:06.504+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:12:06.525 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:06.524+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:12:06.531 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:06.528+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:12:06.532 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:06.528+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:12:24.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:12:25.094 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:12:25.095 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:12:25.162 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2069,"stamp":"2023-10-21T05:12:23.640183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21351765859,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":883919,"num_read_kb":3416671,"num_write":869122,"num_write_kb":3476979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62229,"ondisk_log_size":62229,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":34000016,"kb_used_data":31292656,"kb_used_omap":36,"kb_used_meta":2707291,"kb_avail":249115504,"statfs":{"total":289910292480,"available":255094276096,"internally_reserved":0,"allocated":32043679744,"data_stored":32042592420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":2772266444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,3,3,9,25,10,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":90112,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1750,"num_read_kb":6932,"num_write":1827,"num_write_kb":7308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002020"},"pg_stats":[{"pgid":"2.7","version":"54'4378","reported_seq":7058,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:18.914827+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:12:18.914827+0000","last_peered":"2023-10-21T05:12:18.914827+0000","last_clean":"2023-10-21T05:12:18.914827+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:12:18.914827+0000","last_undegraded":"2023-10-21T05:12:18.914827+0000","last_fullsized":"2023-10-21T05:12:18.914827+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7258","reported_seq":11880,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:18.955947+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:12:18.955947+0000","last_peered":"2023-10-21T05:12:18.955947+0000","last_clean":"2023-10-21T05:12:18.955947+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:12:18.955947+0000","last_undegraded":"2023-10-21T05:12:18.955947+0000","last_fullsized":"2023-10-21T05:12:18.955947+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7258,"log_dups_size":0,"ondisk_log_size":7258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1483","reported_seq":2454,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:15.273552+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:12:15.273552+0000","last_peered":"2023-10-21T05:12:15.273552+0000","last_clean":"2023-10-21T05:12:15.273552+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:12:15.273552+0000","last_undegraded":"2023-10-21T05:12:15.273552+0000","last_fullsized":"2023-10-21T05:12:15.273552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1483,"log_dups_size":0,"ondisk_log_size":1483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1502","reported_seq":2594,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:15.274437+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:12:15.274437+0000","last_peered":"2023-10-21T05:12:15.274437+0000","last_clean":"2023-10-21T05:12:15.274437+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:12:15.274437+0000","last_undegraded":"2023-10-21T05:12:15.274437+0000","last_fullsized":"2023-10-21T05:12:15.274437+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1502,"log_dups_size":0,"ondisk_log_size":1502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'225505","reported_seq":452709,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:21.248728+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:12:21.248728+0000","last_peered":"2023-10-21T05:12:21.248728+0000","last_clean":"2023-10-21T05:12:21.248728+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:12:21.248728+0000","last_undegraded":"2023-10-21T05:12:21.248728+0000","last_fullsized":"2023-10-21T05:12:21.248728+0000","mapping_epoch":17,"log_start":"52'215481","ondisk_log_start":"52'215481","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":149,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 57s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5520433152,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":227078,"num_read_kb":882156,"num_write":225505,"num_write_kb":902020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1575","reported_seq":2557,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:20.393511+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:12:20.393511+0000","last_peered":"2023-10-21T05:12:20.393511+0000","last_clean":"2023-10-21T05:12:20.393511+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:12:20.393511+0000","last_undegraded":"2023-10-21T05:12:20.393511+0000","last_fullsized":"2023-10-21T05:12:20.393511+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1575,"log_dups_size":0,"ondisk_log_size":1575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'210401","reported_seq":423506,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:19.413727+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:12:19.413727+0000","last_peered":"2023-10-21T05:12:19.413727+0000","last_clean":"2023-10-21T05:12:19.413727+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:12:19.413727+0000","last_undegraded":"2023-10-21T05:12:19.413727+0000","last_fullsized":"2023-10-21T05:12:19.413727+0000","mapping_epoch":17,"log_start":"52'200397","ondisk_log_start":"52'200397","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":141,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 44s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5188681728,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":212977,"num_read_kb":827000,"num_write":210401,"num_write_kb":841604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":982,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:15.287922+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:12:15.287922+0000","last_peered":"2023-10-21T05:12:15.287922+0000","last_clean":"2023-10-21T05:12:15.287922+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:12:15.287922+0000","last_undegraded":"2023-10-21T05:12:15.287922+0000","last_fullsized":"2023-10-21T05:12:15.287922+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'218925","reported_seq":440305,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:21.248510+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:12:21.248510+0000","last_peered":"2023-10-21T05:12:21.248510+0000","last_clean":"2023-10-21T05:12:21.248510+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:12:21.248510+0000","last_undegraded":"2023-10-21T05:12:21.248510+0000","last_fullsized":"2023-10-21T05:12:21.248510+0000","mapping_epoch":17,"log_start":"52'208921","ondisk_log_start":"52'208921","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":154,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 56s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5383094272,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":221246,"num_read_kb":860036,"num_write":218925,"num_write_kb":875700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3619,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:20.396766+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:12:20.396766+0000","last_peered":"2023-10-21T05:12:20.396766+0000","last_clean":"2023-10-21T05:12:20.396766+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:12:20.396766+0000","last_undegraded":"2023-10-21T05:12:20.396766+0000","last_fullsized":"2023-10-21T05:12:20.396766+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'213891","reported_seq":430848,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:19.429713+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:12:19.429713+0000","last_peered":"2023-10-21T05:12:19.429713+0000","last_clean":"2023-10-21T05:12:19.429713+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:12:19.429713+0000","last_undegraded":"2023-10-21T05:12:19.429713+0000","last_fullsized":"2023-10-21T05:12:19.429713+0000","mapping_epoch":17,"log_start":"52'203861","ondisk_log_start":"52'203861","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5258965011,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":216852,"num_read_kb":842572,"num_write":213891,"num_write_kb":855558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5782","reported_seq":9294,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:18.971895+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:12:18.971895+0000","last_peered":"2023-10-21T05:12:18.971895+0000","last_clean":"2023-10-21T05:12:18.971895+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:12:18.971895+0000","last_undegraded":"2023-10-21T05:12:18.971895+0000","last_fullsized":"2023-10-21T05:12:18.971895+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":5782,"log_dups_size":0,"ondisk_log_size":5782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":224,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:22.695206+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:22.695206+0000","last_peered":"2023-10-21T05:11:22.695206+0000","last_clean":"2023-10-21T05:11:22.695206+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:22.695206+0000","last_undegraded":"2023-10-21T05:11:22.695206+0000","last_fullsized":"2023-10-21T05:11:22.695206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21351174163,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":878153,"num_read_kb":3411764,"num_write":868722,"num_write_kb":3474882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32040833024,"data_stored":32040833024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40062,"ondisk_log_size":40062,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":22068,"ondisk_log_size":22068,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739175,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11314168,"kb_used_data":10430448,"kb_used_omap":8,"kb_used_meta":883703,"kb_avail":83057672,"statfs":{"total":96636764160,"available":85051056128,"internally_reserved":0,"allocated":10680778752,"data_stored":10680474221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":904912541},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.98999999999999999,"15min":0.92800000000000005},"min":{"1min":0.18099999999999999,"5min":0.17499999999999999,"15min":0.14699999999999999},"max":{"1min":3.7160000000000002,"5min":22.969000000000001,"15min":22.969000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.85199999999999998,"15min":0.69599999999999995},"min":{"1min":0.20699999999999999,"5min":0.20200000000000001,"15min":0.16500000000000001},"max":{"1min":4.391,"5min":22.835000000000001,"15min":22.835000000000001},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Oct 21 05:11:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.51300000000000001,"15min":0.56399999999999995},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.184},"max":{"1min":3.7669999999999999,"5min":4.5309999999999997,"15min":7.8399999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.302,"5min":1.0780000000000001,"15min":0.86599999999999999},"min":{"1min":0.251,"5min":0.19700000000000001,"15min":0.19600000000000001},"max":{"1min":5.2290000000000001,"5min":6.5410000000000004,"15min":8.5150000000000006},"last":0.376}]}]},{"osd":1,"up_from":8,"seq":34359739173,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11349040,"kb_used_data":10431136,"kb_used_omap":15,"kb_used_meta":917872,"kb_avail":83022800,"statfs":{"total":96636764160,"available":85015347200,"internally_reserved":0,"allocated":10681483264,"data_stored":10681065468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":939901318},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,0,5,6,4,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.753,"15min":0.79000000000000004},"min":{"1min":0.157,"5min":0.13,"15min":0.13},"max":{"1min":2.3010000000000002,"5min":10.627000000000001,"15min":11.225},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.313,"5min":0.46500000000000002,"15min":0.47999999999999998},"min":{"1min":0.20300000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.55000000000000004,"5min":5.3209999999999997,"15min":7.1639999999999997},"last":0.28499999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.80800000000000005,"15min":0.78100000000000003},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":3.4900000000000002,"5min":6.6180000000000003,"15min":8.5389999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.75900000000000001,"15min":0.80100000000000005},"min":{"1min":0.22600000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.5259999999999998,"5min":6.1669999999999998,"15min":9.1950000000000003},"last":0.372}]}]},{"osd":0,"up_from":8,"seq":34359739175,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11336808,"kb_used_data":10431072,"kb_used_omap":12,"kb_used_meta":905715,"kb_avail":83035032,"statfs":{"total":96636764160,"available":85027872768,"internally_reserved":0,"allocated":10681417728,"data_stored":10681052731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":927452585},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,15,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:11:25 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.57999999999999996,"15min":0.54500000000000004},"min":{"1min":0.216,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":0.626,"5min":6.5780000000000003,"15min":11.773},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.67500000000000004,"15min":0.60599999999999998},"min":{"1min":0.24399999999999999,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":0.94799999999999995,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.443}]},{"osd":2,"last update":"Sat Oct 21 05:11:25 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.77100000000000002,"15min":0.63600000000000001},"min":{"1min":0.25600000000000001,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":3.863,"5min":6.7409999999999997,"15min":6.7409999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.3280000000000001,"5min":0.99099999999999999,"15min":0.84599999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":5.625,"5min":7.8710000000000004,"15min":10.074999999999999},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10680250368,"data_stored":10680250368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10680295424,"data_stored":10680295424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10680287232,"data_stored":10680287232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:12:25.164 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:25.165 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:25.165 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:25.166 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:25.166 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:12:34.660 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T05:12:34.660 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:12:34.661 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T05:12:34.661 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:12:34.661 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T05:12:34.661 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:12:34.662 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:12:34.662 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:12:34.662 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:12:34.663 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T05:12:34.663 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T05:12:34.663 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T05:12:34.663 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:12:34.664 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:12:34.655+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:12:45.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:12:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:12:45.974 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:12:46.013 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2080,"stamp":"2023-10-21T05:12:45.644288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21352449891,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":887984,"num_read_kb":3432783,"num_write":873282,"num_write_kb":3493619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62421,"ondisk_log_size":62421,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":34095344,"kb_used_data":31293636,"kb_used_omap":36,"kb_used_meta":2801627,"kb_avail":249020176,"statfs":{"total":289910292480,"available":254996660224,"internally_reserved":0,"allocated":32044683264,"data_stored":32043595940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":2868866508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,9,16,17,11,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":471040,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2334,"num_read_kb":9252,"num_write":2319,"num_write_kb":9276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002045"},"pg_stats":[{"pgid":"2.7","version":"54'4393","reported_seq":7088,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:44.272177+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:12:44.272177+0000","last_peered":"2023-10-21T05:12:44.272177+0000","last_clean":"2023-10-21T05:12:44.272177+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:12:44.272177+0000","last_undegraded":"2023-10-21T05:12:44.272177+0000","last_fullsized":"2023-10-21T05:12:44.272177+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7283","reported_seq":11930,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:44.316777+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:12:44.316777+0000","last_peered":"2023-10-21T05:12:44.316777+0000","last_clean":"2023-10-21T05:12:44.316777+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:12:44.316777+0000","last_undegraded":"2023-10-21T05:12:44.316777+0000","last_fullsized":"2023-10-21T05:12:44.316777+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7283,"log_dups_size":0,"ondisk_log_size":7283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1488","reported_seq":2465,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:44.264986+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:12:44.264986+0000","last_peered":"2023-10-21T05:12:44.264986+0000","last_clean":"2023-10-21T05:12:44.264986+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:12:44.264986+0000","last_undegraded":"2023-10-21T05:12:44.264986+0000","last_fullsized":"2023-10-21T05:12:44.264986+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1488,"log_dups_size":0,"ondisk_log_size":1488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1507","reported_seq":2605,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:42.157932+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:12:42.157932+0000","last_peered":"2023-10-21T05:12:42.157932+0000","last_clean":"2023-10-21T05:12:42.157932+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:12:42.157932+0000","last_undegraded":"2023-10-21T05:12:42.157932+0000","last_fullsized":"2023-10-21T05:12:42.157932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'226449","reported_seq":454616,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:41.490870+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:12:41.490870+0000","last_peered":"2023-10-21T05:12:41.490870+0000","last_clean":"2023-10-21T05:12:41.490870+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:12:41.490870+0000","last_undegraded":"2023-10-21T05:12:41.490870+0000","last_fullsized":"2023-10-21T05:12:41.490870+0000","mapping_epoch":17,"log_start":"52'216392","ondisk_log_start":"52'216392","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":239,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 77s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5520502784,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":228028,"num_read_kb":885916,"num_write":226449,"num_write_kb":905796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1584","reported_seq":2570,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:39.221971+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:12:39.221971+0000","last_peered":"2023-10-21T05:12:39.221971+0000","last_clean":"2023-10-21T05:12:39.221971+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:12:39.221971+0000","last_undegraded":"2023-10-21T05:12:39.221971+0000","last_fullsized":"2023-10-21T05:12:39.221971+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'211565","reported_seq":425781,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:44.789679+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:12:44.789679+0000","last_peered":"2023-10-21T05:12:44.789679+0000","last_clean":"2023-10-21T05:12:44.789679+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:12:44.789679+0000","last_undegraded":"2023-10-21T05:12:44.789679+0000","last_fullsized":"2023-10-21T05:12:44.789679+0000","mapping_epoch":17,"log_start":"52'201513","ondisk_log_start":"52'201513","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":209,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 69s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5188825088,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":214074,"num_read_kb":831344,"num_write":211565,"num_write_kb":846260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":986,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:35.756872+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:12:35.756872+0000","last_peered":"2023-10-21T05:12:35.756872+0000","last_clean":"2023-10-21T05:12:35.756872+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:12:35.756872+0000","last_undegraded":"2023-10-21T05:12:35.756872+0000","last_fullsized":"2023-10-21T05:12:35.756872+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'219837","reported_seq":442146,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:41.486911+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:12:41.486911+0000","last_peered":"2023-10-21T05:12:41.486911+0000","last_clean":"2023-10-21T05:12:41.486911+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:12:41.486911+0000","last_undegraded":"2023-10-21T05:12:41.486911+0000","last_fullsized":"2023-10-21T05:12:41.486911+0000","mapping_epoch":17,"log_start":"52'209831","ondisk_log_start":"52'209831","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":211,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 76s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5383245824,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":222169,"num_read_kb":863696,"num_write":219837,"num_write_kb":879348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3635,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:40.900061+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:12:40.900061+0000","last_peered":"2023-10-21T05:12:40.900061+0000","last_clean":"2023-10-21T05:12:40.900061+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:12:40.900061+0000","last_undegraded":"2023-10-21T05:12:40.900061+0000","last_fullsized":"2023-10-21T05:12:40.900061+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'215031","reported_seq":433084,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:44.787509+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:12:44.787509+0000","last_peered":"2023-10-21T05:12:44.787509+0000","last_clean":"2023-10-21T05:12:44.787509+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:12:44.787509+0000","last_undegraded":"2023-10-21T05:12:44.787509+0000","last_fullsized":"2023-10-21T05:12:44.787509+0000","mapping_epoch":17,"log_start":"52'204971","ondisk_log_start":"52'204971","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5259284499,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":217947,"num_read_kb":846920,"num_write":215031,"num_write_kb":860118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5802","reported_seq":9335,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:44.315141+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:12:44.315141+0000","last_peered":"2023-10-21T05:12:44.315141+0000","last_clean":"2023-10-21T05:12:44.315141+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:12:44.315141+0000","last_undegraded":"2023-10-21T05:12:44.315141+0000","last_fullsized":"2023-10-21T05:12:44.315141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":5802,"log_dups_size":0,"ondisk_log_size":5802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":224,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:22.695206+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:22.695206+0000","last_peered":"2023-10-21T05:11:22.695206+0000","last_clean":"2023-10-21T05:11:22.695206+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:22.695206+0000","last_undegraded":"2023-10-21T05:11:22.695206+0000","last_fullsized":"2023-10-21T05:11:22.695206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21351858195,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":882218,"num_read_kb":3427876,"num_write":872882,"num_write_kb":3491522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32041836544,"data_stored":32041836544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40175,"ondisk_log_size":40175,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":22147,"ondisk_log_size":22147,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739179,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11350584,"kb_used_data":10430768,"kb_used_omap":8,"kb_used_meta":919799,"kb_avail":83021256,"statfs":{"total":96636764160,"available":85013766144,"internally_reserved":0,"allocated":10681106432,"data_stored":10680801901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":941874845},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,4,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:40 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.97699999999999998,"15min":0.92000000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.14699999999999999},"max":{"1min":3.5379999999999998,"5min":22.969000000000001,"15min":22.969000000000001},"last":0.37},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.86299999999999999,"15min":0.71799999999999997},"min":{"1min":0.20999999999999999,"5min":0.20699999999999999,"15min":0.16500000000000001},"max":{"1min":5.1689999999999996,"5min":22.835000000000001,"15min":22.835000000000001},"last":0.38900000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":0.75800000000000001,"15min":0.64600000000000002},"min":{"1min":0.20100000000000001,"5min":0.19,"15min":0.184},"max":{"1min":19.574000000000002,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.1319999999999999,"15min":0.93799999999999994},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":21.234999999999999,"5min":21.234999999999999,"15min":21.234999999999999},"last":0.54500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739178,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11389576,"kb_used_data":10431468,"kb_used_omap":15,"kb_used_meta":958064,"kb_avail":82982264,"statfs":{"total":96636764160,"available":84973838336,"internally_reserved":0,"allocated":10681823232,"data_stored":10681405436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":981057926},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,10,9,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.75800000000000001,"15min":0.79000000000000004},"min":{"1min":0.13300000000000001,"5min":0.13,"15min":0.13},"max":{"1min":3.4180000000000001,"5min":10.627000000000001,"15min":11.225},"last":0.16500000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.50700000000000001,"15min":0.50800000000000001},"min":{"1min":0.21099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":9.1210000000000004,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.27600000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:12:19 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":0.91900000000000004,"15min":0.81000000000000005},"min":{"1min":0.26700000000000002,"5min":0.185,"15min":0.185},"max":{"1min":5.7039999999999997,"5min":6.6180000000000003,"15min":8.5389999999999997},"last":0.316},{"interface":"front","average":{"1min":1.3120000000000001,"5min":0.874,"15min":0.83099999999999996},"min":{"1min":0.32400000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":5.7400000000000002,"5min":6.1669999999999998,"15min":9.1950000000000003},"last":0.34200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739179,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11355184,"kb_used_data":10431400,"kb_used_omap":12,"kb_used_meta":923763,"kb_avail":83016656,"statfs":{"total":96636764160,"available":85009055744,"internally_reserved":0,"allocated":10681753600,"data_stored":10681388603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":945933737},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,4,3,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.189,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":5.4240000000000004,"5min":6.5759999999999996,"15min":11.773},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.60399999999999998,"15min":0.61399999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":5.4560000000000004,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.36099999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.83599999999999997,"15min":0.63300000000000001},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":9.3140000000000001,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.878,"5min":1.0409999999999999,"15min":0.85199999999999998},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.516,"5min":9.516,"15min":10.074999999999999},"last":0.59899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10680602624,"data_stored":10680602624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10680659968,"data_stored":10680659968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10680573952,"data_stored":10680573952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:12:46.016 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:46.016 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:46.016 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:46.017 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:46.017 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:13:05.420 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T05:13:05.414+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T05:13:06.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:13:07.077 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:13:07.078 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:13:07.130 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2090,"stamp":"2023-10-21T05:13:05.653298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21353351011,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":891745,"num_read_kb":3447723,"num_write":876927,"num_write_kb":3508199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62398,"ondisk_log_size":62398,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":34258844,"kb_used_data":31295108,"kb_used_omap":36,"kb_used_meta":2963675,"kb_avail":248856676,"statfs":{"total":289910292480,"available":254829236224,"internally_reserved":0,"allocated":32046190592,"data_stored":32045103268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":3034803660},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,9,8,32,13,12,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":630784,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2369,"num_read_kb":9412,"num_write":2243,"num_write_kb":8972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007554"},"pg_stats":[{"pgid":"2.7","version":"54'4408","reported_seq":7116,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:02.707402+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:13:02.707402+0000","last_peered":"2023-10-21T05:13:02.707402+0000","last_clean":"2023-10-21T05:13:02.707402+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:13:02.707402+0000","last_undegraded":"2023-10-21T05:13:02.707402+0000","last_fullsized":"2023-10-21T05:13:02.707402+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7308","reported_seq":11975,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:02.711006+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:13:02.711006+0000","last_peered":"2023-10-21T05:13:02.711006+0000","last_clean":"2023-10-21T05:13:02.711006+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:13:02.711006+0000","last_undegraded":"2023-10-21T05:13:02.711006+0000","last_fullsized":"2023-10-21T05:13:02.711006+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7308,"log_dups_size":0,"ondisk_log_size":7308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1493","reported_seq":2474,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:01.354162+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:13:01.354162+0000","last_peered":"2023-10-21T05:13:01.354162+0000","last_clean":"2023-10-21T05:13:01.354162+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:13:01.354162+0000","last_undegraded":"2023-10-21T05:13:01.354162+0000","last_fullsized":"2023-10-21T05:13:01.354162+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1493,"log_dups_size":0,"ondisk_log_size":1493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1512","reported_seq":2614,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:01.354417+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:13:01.354417+0000","last_peered":"2023-10-21T05:13:01.354417+0000","last_clean":"2023-10-21T05:13:01.354417+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:13:01.354417+0000","last_undegraded":"2023-10-21T05:13:01.354417+0000","last_fullsized":"2023-10-21T05:13:01.354417+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1512,"log_dups_size":0,"ondisk_log_size":1512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'227452","reported_seq":456528,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:01.698304+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:13:01.698304+0000","last_peered":"2023-10-21T05:13:01.698304+0000","last_clean":"2023-10-21T05:13:01.698304+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:13:01.698304+0000","last_undegraded":"2023-10-21T05:13:01.698304+0000","last_fullsized":"2023-10-21T05:13:01.698304+0000","mapping_epoch":17,"log_start":"53'217403","ondisk_log_start":"53'217403","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":272,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 97s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5520887808,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":228929,"num_read_kb":889508,"num_write":227452,"num_write_kb":909808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1590","reported_seq":2580,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:01.353795+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:13:01.353795+0000","last_peered":"2023-10-21T05:13:01.353795+0000","last_clean":"2023-10-21T05:13:01.353795+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:13:01.353795+0000","last_undegraded":"2023-10-21T05:13:01.353795+0000","last_fullsized":"2023-10-21T05:13:01.353795+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1590,"log_dups_size":0,"ondisk_log_size":1590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'212425","reported_seq":427572,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:05.003244+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:13:05.003244+0000","last_peered":"2023-10-21T05:13:05.003244+0000","last_clean":"2023-10-21T05:13:05.003244+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:13:05.003244+0000","last_undegraded":"2023-10-21T05:13:05.003244+0000","last_fullsized":"2023-10-21T05:13:05.003244+0000","mapping_epoch":17,"log_start":"52'202423","ondisk_log_start":"52'202423","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":284,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 90s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5188980736,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":214994,"num_read_kb":834996,"num_write":212425,"num_write_kb":849700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":990,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:12:56.244510+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:12:56.244510+0000","last_peered":"2023-10-21T05:12:56.244510+0000","last_clean":"2023-10-21T05:12:56.244510+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:12:56.244510+0000","last_undegraded":"2023-10-21T05:12:56.244510+0000","last_fullsized":"2023-10-21T05:12:56.244510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'220756","reported_seq":444059,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:01.767356+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:13:01.767356+0000","last_peered":"2023-10-21T05:13:01.767356+0000","last_clean":"2023-10-21T05:13:01.767356+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:13:01.767356+0000","last_undegraded":"2023-10-21T05:13:01.767356+0000","last_fullsized":"2023-10-21T05:13:01.767356+0000","mapping_epoch":17,"log_start":"52'210748","ondisk_log_start":"52'210748","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":265,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 96s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5383487488,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":223150,"num_read_kb":867596,"num_write":220756,"num_write_kb":883024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3651,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:01.365428+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:13:01.365428+0000","last_peered":"2023-10-21T05:13:01.365428+0000","last_clean":"2023-10-21T05:13:01.365428+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:13:01.365428+0000","last_undegraded":"2023-10-21T05:13:01.365428+0000","last_fullsized":"2023-10-21T05:13:01.365428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'215894","reported_seq":434906,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:05.016157+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:13:05.016157+0000","last_peered":"2023-10-21T05:13:05.016157+0000","last_clean":"2023-10-21T05:13:05.016157+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:13:05.016157+0000","last_undegraded":"2023-10-21T05:13:05.016157+0000","last_fullsized":"2023-10-21T05:13:05.016157+0000","mapping_epoch":17,"log_start":"52'205877","ondisk_log_start":"52'205877","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5259403283,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":218906,"num_read_kb":850716,"num_write":215894,"num_write_kb":863570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5822","reported_seq":9371,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:02.793030+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:13:02.793030+0000","last_peered":"2023-10-21T05:13:02.793030+0000","last_clean":"2023-10-21T05:13:02.793030+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:13:02.793030+0000","last_undegraded":"2023-10-21T05:13:02.793030+0000","last_fullsized":"2023-10-21T05:13:02.793030+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":5822,"log_dups_size":0,"ondisk_log_size":5822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":224,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:22.695206+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:22.695206+0000","last_peered":"2023-10-21T05:11:22.695206+0000","last_clean":"2023-10-21T05:11:22.695206+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:22.695206+0000","last_undegraded":"2023-10-21T05:11:22.695206+0000","last_fullsized":"2023-10-21T05:11:22.695206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21352759315,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":885979,"num_read_kb":3442816,"num_write":876527,"num_write_kb":3506102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32043282432,"data_stored":32043282432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40076,"ondisk_log_size":40076,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":22223,"ondisk_log_size":22223,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739183,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11427036,"kb_used_data":10431252,"kb_used_omap":8,"kb_used_meta":995767,"kb_avail":82944804,"statfs":{"total":96636764160,"available":84935479296,"internally_reserved":0,"allocated":10681602048,"data_stored":10681297517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1019666077},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,7,5,9,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:40 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.97699999999999998,"15min":0.92000000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.14699999999999999},"max":{"1min":3.5379999999999998,"5min":22.969000000000001,"15min":22.969000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.86299999999999999,"15min":0.71799999999999997},"min":{"1min":0.20999999999999999,"5min":0.20699999999999999,"15min":0.16500000000000001},"max":{"1min":5.1689999999999996,"5min":22.835000000000001,"15min":22.835000000000001},"last":0.32800000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":0.75800000000000001,"15min":0.64600000000000002},"min":{"1min":0.20100000000000001,"5min":0.19,"15min":0.184},"max":{"1min":19.574000000000002,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.1319999999999999,"15min":0.93799999999999994},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":21.234999999999999,"5min":21.234999999999999,"15min":21.234999999999999},"last":0.42599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739182,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11440052,"kb_used_data":10431980,"kb_used_omap":15,"kb_used_meta":1008048,"kb_avail":82931788,"statfs":{"total":96636764160,"available":84922150912,"internally_reserved":0,"allocated":10682347520,"data_stored":10681929724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1032241542},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.75800000000000001,"15min":0.79000000000000004},"min":{"1min":0.13300000000000001,"5min":0.13,"15min":0.13},"max":{"1min":3.4180000000000001,"5min":10.627000000000001,"15min":11.225},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.50700000000000001,"15min":0.50800000000000001},"min":{"1min":0.21099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":9.1210000000000004,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:12:19 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":0.91900000000000004,"15min":0.81000000000000005},"min":{"1min":0.26700000000000002,"5min":0.185,"15min":0.185},"max":{"1min":5.7039999999999997,"5min":6.6180000000000003,"15min":8.5389999999999997},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.3120000000000001,"5min":0.874,"15min":0.83099999999999996},"min":{"1min":0.32400000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":5.7400000000000002,"5min":6.1669999999999998,"15min":9.1950000000000003},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739183,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11391756,"kb_used_data":10431876,"kb_used_omap":12,"kb_used_meta":959859,"kb_avail":82980084,"statfs":{"total":96636764160,"available":84971606016,"internally_reserved":0,"allocated":10682241024,"data_stored":10681876027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":982896041},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,19,7,6,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.189,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":5.4240000000000004,"5min":6.5759999999999996,"15min":11.773},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.60399999999999998,"15min":0.61399999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":5.4560000000000004,"5min":6.6429999999999998,"15min":11.808999999999999},"last":1.393}]},{"osd":2,"last update":"Sat Oct 21 05:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.83599999999999997,"15min":0.63300000000000001},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":9.3140000000000001,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.879},{"interface":"front","average":{"1min":0.878,"5min":1.0409999999999999,"15min":0.85199999999999998},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.516,"5min":9.516,"15min":10.074999999999999},"last":0.85999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10681094144,"data_stored":10681094144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10681114624,"data_stored":10681114624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10681073664,"data_stored":10681073664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:13:07.131 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:07.132 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:07.132 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:07.133 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:07.133 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:13:13.520 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:13.510+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:13:13.521 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:13.514+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:13:13.522 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:13.514+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:13:13.523 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:13.514+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:13:13.523 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:13.514+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:13:13.523 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:13.514+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:13:13.523 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:13.514+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:13:18.081 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:18.066+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:13:18.082 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:18.066+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:13:18.083 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:18.066+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:13:18.089 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:18.078+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:13:27.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:13:27.744 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:13:27.745 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:13:27.817 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2101,"stamp":"2023-10-21T05:13:27.657738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21354026851,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":895920,"num_read_kb":3464327,"num_write":881052,"num_write_kb":3524699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62759,"ondisk_log_size":62759,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":34288864,"kb_used_data":31296136,"kb_used_omap":36,"kb_used_meta":2992667,"kb_avail":248826656,"statfs":{"total":289910292480,"available":254798495744,"internally_reserved":0,"allocated":32047243264,"data_stored":32046155940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":3064491468},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,2,7,47,14,13,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":389120,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2275,"num_read_kb":9040,"num_write":2316,"num_write_kb":9264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002209"},"pg_stats":[{"pgid":"2.7","version":"54'4448","reported_seq":7168,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:21.833599+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:13:21.833599+0000","last_peered":"2023-10-21T05:13:21.833599+0000","last_clean":"2023-10-21T05:13:21.833599+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:13:21.833599+0000","last_undegraded":"2023-10-21T05:13:21.833599+0000","last_fullsized":"2023-10-21T05:13:21.833599+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7374","reported_seq":12062,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:21.834259+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:13:21.834259+0000","last_peered":"2023-10-21T05:13:21.834259+0000","last_clean":"2023-10-21T05:13:21.834259+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:13:21.834259+0000","last_undegraded":"2023-10-21T05:13:21.834259+0000","last_fullsized":"2023-10-21T05:13:21.834259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7374,"log_dups_size":0,"ondisk_log_size":7374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1507","reported_seq":2492,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:18.082187+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:13:18.082187+0000","last_peered":"2023-10-21T05:13:18.082187+0000","last_clean":"2023-10-21T05:13:18.082187+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:13:18.082187+0000","last_undegraded":"2023-10-21T05:13:18.082187+0000","last_fullsized":"2023-10-21T05:13:18.082187+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1525","reported_seq":2631,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:18.081502+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:13:18.081502+0000","last_peered":"2023-10-21T05:13:18.081502+0000","last_clean":"2023-10-21T05:13:18.081502+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:13:18.081502+0000","last_undegraded":"2023-10-21T05:13:18.081502+0000","last_fullsized":"2023-10-21T05:13:18.081502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1525,"log_dups_size":0,"ondisk_log_size":1525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'228618","reported_seq":458860,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:26.885131+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:13:26.885131+0000","last_peered":"2023-10-21T05:13:26.885131+0000","last_clean":"2023-10-21T05:13:26.885131+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:13:26.885131+0000","last_undegraded":"2023-10-21T05:13:26.885131+0000","last_fullsized":"2023-10-21T05:13:26.885131+0000","mapping_epoch":17,"log_start":"53'218519","ondisk_log_start":"53'218519","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":344,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 122s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5521055744,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":230087,"num_read_kb":894112,"num_write":228618,"num_write_kb":914472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1601","reported_seq":2595,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:22.765240+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:13:22.765240+0000","last_peered":"2023-10-21T05:13:22.765240+0000","last_clean":"2023-10-21T05:13:22.765240+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:13:22.765240+0000","last_undegraded":"2023-10-21T05:13:22.765240+0000","last_fullsized":"2023-10-21T05:13:22.765240+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1601,"log_dups_size":0,"ondisk_log_size":1601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'213357","reported_seq":429404,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:25.250076+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:13:25.250076+0000","last_peered":"2023-10-21T05:13:25.250076+0000","last_clean":"2023-10-21T05:13:25.250076+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:13:25.250076+0000","last_undegraded":"2023-10-21T05:13:25.250076+0000","last_fullsized":"2023-10-21T05:13:25.250076+0000","mapping_epoch":17,"log_start":"53'203334","ondisk_log_start":"53'203334","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":320,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 110s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5189263360,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":215889,"num_read_kb":838548,"num_write":213357,"num_write_kb":853428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":994,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:16.723099+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:13:16.723099+0000","last_peered":"2023-10-21T05:13:16.723099+0000","last_clean":"2023-10-21T05:13:16.723099+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:13:16.723099+0000","last_undegraded":"2023-10-21T05:13:16.723099+0000","last_fullsized":"2023-10-21T05:13:16.723099+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'221919","reported_seq":446461,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:26.907231+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:13:26.907231+0000","last_peered":"2023-10-21T05:13:26.907231+0000","last_clean":"2023-10-21T05:13:26.907231+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:13:26.907231+0000","last_undegraded":"2023-10-21T05:13:26.907231+0000","last_fullsized":"2023-10-21T05:13:26.907231+0000","mapping_epoch":17,"log_start":"53'211875","ondisk_log_start":"53'211875","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":318,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 121s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5383622656,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":224380,"num_read_kb":872500,"num_write":221919,"num_write_kb":887676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3667,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:21.836145+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:13:21.836145+0000","last_peered":"2023-10-21T05:13:21.836145+0000","last_clean":"2023-10-21T05:13:21.836145+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:13:21.836145+0000","last_undegraded":"2023-10-21T05:13:21.836145+0000","last_fullsized":"2023-10-21T05:13:21.836145+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'216758","reported_seq":436658,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:25.207399+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:13:25.207399+0000","last_peered":"2023-10-21T05:13:25.207399+0000","last_clean":"2023-10-21T05:13:25.207399+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:13:25.207399+0000","last_undegraded":"2023-10-21T05:13:25.207399+0000","last_fullsized":"2023-10-21T05:13:25.207399+0000","mapping_epoch":17,"log_start":"53'206682","ondisk_log_start":"53'206682","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5259493395,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":219798,"num_read_kb":854260,"num_write":216758,"num_write_kb":867026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5873","reported_seq":9438,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:21.837616+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:13:21.837616+0000","last_peered":"2023-10-21T05:13:21.837616+0000","last_clean":"2023-10-21T05:13:21.837616+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:13:21.837616+0000","last_undegraded":"2023-10-21T05:13:21.837616+0000","last_fullsized":"2023-10-21T05:13:21.837616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":5873,"log_dups_size":0,"ondisk_log_size":5873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":224,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:22.695206+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:22.695206+0000","last_peered":"2023-10-21T05:11:22.695206+0000","last_clean":"2023-10-21T05:11:22.695206+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:22.695206+0000","last_undegraded":"2023-10-21T05:11:22.695206+0000","last_fullsized":"2023-10-21T05:11:22.695206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21353435155,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":890154,"num_read_kb":3459420,"num_write":880652,"num_write_kb":3522602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32044437504,"data_stored":32044437504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40242,"ondisk_log_size":40242,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":22418,"ondisk_log_size":22418,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739187,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11420288,"kb_used_data":10431608,"kb_used_omap":8,"kb_used_meta":988663,"kb_avail":82951552,"statfs":{"total":96636764160,"available":84942389248,"internally_reserved":0,"allocated":10681966592,"data_stored":10681662061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1012391581},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,12,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:40 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.97699999999999998,"15min":0.92000000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.14699999999999999},"max":{"1min":3.5379999999999998,"5min":22.969000000000001,"15min":22.969000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.86299999999999999,"15min":0.71799999999999997},"min":{"1min":0.20999999999999999,"5min":0.20699999999999999,"15min":0.16500000000000001},"max":{"1min":5.1689999999999996,"5min":22.835000000000001,"15min":22.835000000000001},"last":0.33000000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":0.75800000000000001,"15min":0.64600000000000002},"min":{"1min":0.20100000000000001,"5min":0.19,"15min":0.184},"max":{"1min":19.574000000000002,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.374},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.1319999999999999,"15min":0.93799999999999994},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":21.234999999999999,"5min":21.234999999999999,"15min":21.234999999999999},"last":0.442}]}]},{"osd":1,"up_from":8,"seq":34359739186,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11440356,"kb_used_data":10432284,"kb_used_omap":15,"kb_used_meta":1008048,"kb_avail":82931484,"statfs":{"total":96636764160,"available":84921839616,"internally_reserved":0,"allocated":10682658816,"data_stored":10682241020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1032241542},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,27,9,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:20 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.69399999999999995,"15min":0.76600000000000001},"min":{"1min":0.16300000000000001,"5min":0.13,"15min":0.13},"max":{"1min":4.4640000000000004,"5min":5.657,"15min":11.225},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.48099999999999998,"15min":0.51500000000000001},"min":{"1min":0.19700000000000001,"5min":0.185,"15min":0.18099999999999999},"max":{"1min":2.1560000000000001,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Oct 21 05:13:20 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.86399999999999999,"15min":0.77400000000000002},"min":{"1min":0.23799999999999999,"5min":0.185,"15min":0.185},"max":{"1min":2.1389999999999998,"5min":6.2060000000000004,"15min":8.5389999999999997},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.86299999999999999,"15min":0.79400000000000004},"min":{"1min":0.28799999999999998,"5min":0.224,"15min":0.20599999999999999},"max":{"1min":2.181,"5min":6.1669999999999998,"15min":9.1950000000000003},"last":0.44900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739188,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11428220,"kb_used_data":10432244,"kb_used_omap":12,"kb_used_meta":995955,"kb_avail":82943620,"statfs":{"total":96636764160,"available":84934266880,"internally_reserved":0,"allocated":10682617856,"data_stored":10682252859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1019858345},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,3,8,0,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:26 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.59299999999999997,"15min":0.56899999999999995},"min":{"1min":0.188,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":5.5709999999999997,"5min":6.5759999999999996,"15min":11.773},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.61799999999999999,"15min":0.64800000000000002},"min":{"1min":0.26200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":5.6100000000000003,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.33800000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":0.88,"15min":0.66400000000000003},"min":{"1min":0.33500000000000002,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":5.6529999999999996,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.377},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.026,"15min":0.85599999999999998},"min":{"1min":0.21199999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":3.8889999999999998,"5min":9.516,"15min":10.074999999999999},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10681483264,"data_stored":10681483264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10681466880,"data_stored":10681466880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10681487360,"data_stored":10681487360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:13:27.819 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:27.819 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:27.819 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:27.820 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:27.820 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:13:45.449 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:45.441+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:13:45.450 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:13:45.441+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:13:47.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:13:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:13:48.680 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:13:48.710 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2111,"stamp":"2023-10-21T05:13:47.661345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21354739555,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":899647,"num_read_kb":3479139,"num_write":884694,"num_write_kb":3539267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62877,"ondisk_log_size":62877,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":34325816,"kb_used_data":31296992,"kb_used_omap":36,"kb_used_meta":3028763,"kb_avail":248789704,"statfs":{"total":289910292480,"available":254760656896,"internally_reserved":0,"allocated":32048119808,"data_stored":32047032484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":3101453772},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,8,16,15,19,14,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":548864,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2295,"num_read_kb":9116,"num_write":2286,"num_write_kb":9144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002156"},"pg_stats":[{"pgid":"2.7","version":"54'4473","reported_seq":7205,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:45.471380+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:13:45.471380+0000","last_peered":"2023-10-21T05:13:45.471380+0000","last_clean":"2023-10-21T05:13:45.471380+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:13:45.471380+0000","last_undegraded":"2023-10-21T05:13:45.471380+0000","last_fullsized":"2023-10-21T05:13:45.471380+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4473,"log_dups_size":0,"ondisk_log_size":4473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7414","reported_seq":12122,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:45.484256+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:13:45.484256+0000","last_peered":"2023-10-21T05:13:45.484256+0000","last_clean":"2023-10-21T05:13:45.484256+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:13:45.484256+0000","last_undegraded":"2023-10-21T05:13:45.484256+0000","last_fullsized":"2023-10-21T05:13:45.484256+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7414,"log_dups_size":0,"ondisk_log_size":7414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1516","reported_seq":2505,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:45.463289+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:13:45.463289+0000","last_peered":"2023-10-21T05:13:45.463289+0000","last_clean":"2023-10-21T05:13:45.463289+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:13:45.463289+0000","last_undegraded":"2023-10-21T05:13:45.463289+0000","last_fullsized":"2023-10-21T05:13:45.463289+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1516,"log_dups_size":0,"ondisk_log_size":1516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1534","reported_seq":2644,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:45.463503+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:13:45.463503+0000","last_peered":"2023-10-21T05:13:45.463503+0000","last_clean":"2023-10-21T05:13:45.463503+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:13:45.463503+0000","last_undegraded":"2023-10-21T05:13:45.463503+0000","last_fullsized":"2023-10-21T05:13:45.463503+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1534,"log_dups_size":0,"ondisk_log_size":1534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'229496","reported_seq":460731,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:47.174330+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:13:47.174330+0000","last_peered":"2023-10-21T05:13:47.174330+0000","last_clean":"2023-10-21T05:13:47.174330+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:13:47.174330+0000","last_undegraded":"2023-10-21T05:13:47.174330+0000","last_fullsized":"2023-10-21T05:13:47.174330+0000","mapping_epoch":17,"log_start":"53'219426","ondisk_log_start":"53'219426","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":429,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 143s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5521424384,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":231067,"num_read_kb":898012,"num_write":229496,"num_write_kb":917984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1606","reported_seq":2604,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:42.317708+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:13:42.317708+0000","last_peered":"2023-10-21T05:13:42.317708+0000","last_clean":"2023-10-21T05:13:42.317708+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:13:42.317708+0000","last_undegraded":"2023-10-21T05:13:42.317708+0000","last_fullsized":"2023-10-21T05:13:42.317708+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1606,"log_dups_size":0,"ondisk_log_size":1606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'214237","reported_seq":431187,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:45.505927+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:13:45.505927+0000","last_peered":"2023-10-21T05:13:45.505927+0000","last_clean":"2023-10-21T05:13:45.505927+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:13:45.505927+0000","last_undegraded":"2023-10-21T05:13:45.505927+0000","last_fullsized":"2023-10-21T05:13:45.505927+0000","mapping_epoch":17,"log_start":"53'204143","ondisk_log_start":"53'204143","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":408,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 130s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5189402624,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":216774,"num_read_kb":842052,"num_write":214237,"num_write_kb":856948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":998,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:37.193547+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:13:37.193547+0000","last_peered":"2023-10-21T05:13:37.193547+0000","last_clean":"2023-10-21T05:13:37.193547+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:13:37.193547+0000","last_undegraded":"2023-10-21T05:13:37.193547+0000","last_fullsized":"2023-10-21T05:13:37.193547+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'222898","reported_seq":448385,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:47.188115+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:13:47.188115+0000","last_peered":"2023-10-21T05:13:47.188115+0000","last_clean":"2023-10-21T05:13:47.188115+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:13:47.188115+0000","last_undegraded":"2023-10-21T05:13:47.188115+0000","last_fullsized":"2023-10-21T05:13:47.188115+0000","mapping_epoch":17,"log_start":"53'212893","ondisk_log_start":"53'212893","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":384,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 142s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5383794688,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":225313,"num_read_kb":876208,"num_write":222898,"num_write_kb":891592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3683,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:42.318468+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:13:42.318468+0000","last_peered":"2023-10-21T05:13:42.318468+0000","last_clean":"2023-10-21T05:13:42.318468+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:13:42.318468+0000","last_undegraded":"2023-10-21T05:13:42.318468+0000","last_fullsized":"2023-10-21T05:13:42.318468+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'217663","reported_seq":438497,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:45.514334+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:13:45.514334+0000","last_peered":"2023-10-21T05:13:45.514334+0000","last_clean":"2023-10-21T05:13:45.514334+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:13:45.514334+0000","last_undegraded":"2023-10-21T05:13:45.514334+0000","last_fullsized":"2023-10-21T05:13:45.514334+0000","mapping_epoch":17,"log_start":"53'207588","ondisk_log_start":"53'207588","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5259526163,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":220727,"num_read_kb":857960,"num_write":217663,"num_write_kb":870646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5901","reported_seq":9482,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:13:45.511705+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:13:45.511705+0000","last_peered":"2023-10-21T05:13:45.511705+0000","last_clean":"2023-10-21T05:13:45.511705+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:13:45.511705+0000","last_undegraded":"2023-10-21T05:13:45.511705+0000","last_fullsized":"2023-10-21T05:13:45.511705+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":5901,"log_dups_size":0,"ondisk_log_size":5901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":224,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:22.695206+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:22.695206+0000","last_peered":"2023-10-21T05:11:22.695206+0000","last_clean":"2023-10-21T05:11:22.695206+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:22.695206+0000","last_undegraded":"2023-10-21T05:11:22.695206+0000","last_fullsized":"2023-10-21T05:11:22.695206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21354147859,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":893881,"num_read_kb":3474232,"num_write":884294,"num_write_kb":3537170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32045305856,"data_stored":32045305856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40244,"ondisk_log_size":40244,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":22534,"ondisk_log_size":22534,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739191,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11438576,"kb_used_data":10431848,"kb_used_omap":8,"kb_used_meta":1006711,"kb_avail":82933264,"statfs":{"total":96636764160,"available":84923662336,"internally_reserved":0,"allocated":10682212352,"data_stored":10681907821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1030872733},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,7,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.81799999999999995,"15min":0.871},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":4.4290000000000003,"5min":7.5579999999999998,"15min":22.969000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.623,"15min":0.70299999999999996},"min":{"1min":0.216,"5min":0.20699999999999999,"15min":0.16500000000000001},"max":{"1min":0.441,"5min":5.1689999999999996,"15min":22.835000000000001},"last":0.26600000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:13:27 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.77300000000000002,"15min":0.629},"min":{"1min":0.219,"5min":0.19,"15min":0.186},"max":{"1min":4.4740000000000002,"5min":19.574000000000002,"15min":19.574000000000002},"last":1.8029999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":1.0589999999999999,"15min":0.94199999999999995},"min":{"1min":0.26000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":3.7909999999999999,"5min":21.234999999999999,"15min":21.234999999999999},"last":1.821}]}]},{"osd":1,"up_from":8,"seq":34359739190,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11458692,"kb_used_data":10432572,"kb_used_omap":15,"kb_used_meta":1026096,"kb_avail":82913148,"statfs":{"total":96636764160,"available":84903063552,"internally_reserved":0,"allocated":10682953728,"data_stored":10682535932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1050722694},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,6,9,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:20 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.69399999999999995,"15min":0.76600000000000001},"min":{"1min":0.16300000000000001,"5min":0.13,"15min":0.13},"max":{"1min":4.4640000000000004,"5min":5.657,"15min":11.225},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.48099999999999998,"15min":0.51500000000000001},"min":{"1min":0.19700000000000001,"5min":0.185,"15min":0.18099999999999999},"max":{"1min":2.1560000000000001,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.30599999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:13:20 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.86399999999999999,"15min":0.77400000000000002},"min":{"1min":0.23799999999999999,"5min":0.185,"15min":0.185},"max":{"1min":2.1389999999999998,"5min":6.2060000000000004,"15min":8.5389999999999997},"last":0.626},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.86299999999999999,"15min":0.79400000000000004},"min":{"1min":0.28799999999999998,"5min":0.224,"15min":0.20599999999999999},"max":{"1min":2.181,"5min":6.1669999999999998,"15min":9.1950000000000003},"last":0.66300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739192,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11428548,"kb_used_data":10432572,"kb_used_omap":12,"kb_used_meta":995955,"kb_avail":82943292,"statfs":{"total":96636764160,"available":84933931008,"internally_reserved":0,"allocated":10682953728,"data_stored":10682588731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1019858345},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,3,11,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:26 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.59299999999999997,"15min":0.56899999999999995},"min":{"1min":0.188,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":5.5709999999999997,"5min":6.5759999999999996,"15min":11.773},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.61799999999999999,"15min":0.64800000000000002},"min":{"1min":0.26200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":5.6100000000000003,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.39200000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":0.88,"15min":0.66400000000000003},"min":{"1min":0.33500000000000002,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":5.6529999999999996,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.026,"15min":0.85599999999999998},"min":{"1min":0.21199999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":3.8889999999999998,"5min":9.516,"15min":10.074999999999999},"last":0.41499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10681827328,"data_stored":10681827328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10681733120,"data_stored":10681733120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10681745408,"data_stored":10681745408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:13:48.712 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:48.712 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:48.713 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:48.713 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:48.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:13:49.488 INFO:teuthology.orchestra.run.smithi172.stderr:instructing pg 3.3s0 on osd.0 to deep-scrub 2023-10-21T05:13:49.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:13:50.321 INFO:teuthology.orchestra.run.smithi172.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2023-10-21T05:13:50.384 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:13:51.241 INFO:teuthology.orchestra.run.smithi172.stderr:instructing pg 3.1s0 on osd.0 to deep-scrub 2023-10-21T05:13:51.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:13:52.023 INFO:teuthology.orchestra.run.smithi172.stderr:instructing pg 3.2s0 on osd.1 to deep-scrub 2023-10-21T05:13:52.099 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:12.101 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:14:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:14:12.828 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:14:12.877 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2123,"stamp":"2023-10-21T05:14:11.665613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21355288419,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":903620,"num_read_kb":3494927,"num_write":888791,"num_write_kb":3555655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62953,"ondisk_log_size":62953,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":34425312,"kb_used_data":31297992,"kb_used_omap":36,"kb_used_meta":3127259,"kb_avail":248690208,"statfs":{"total":289910292480,"available":254658772992,"internally_reserved":0,"allocated":32049143808,"data_stored":32048056484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":3202313676},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,8,11,21,14,25,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":458752,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2193,"num_read_kb":8708,"num_write":2311,"num_write_kb":9244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002099"},"pg_stats":[{"pgid":"2.7","version":"54'4482","reported_seq":7229,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:07.913523+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:14:07.913523+0000","last_peered":"2023-10-21T05:14:07.913523+0000","last_clean":"2023-10-21T05:14:07.913523+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:14:07.913523+0000","last_undegraded":"2023-10-21T05:14:07.913523+0000","last_fullsized":"2023-10-21T05:14:07.913523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7429","reported_seq":12162,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:07.914069+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:14:07.914069+0000","last_peered":"2023-10-21T05:14:07.914069+0000","last_clean":"2023-10-21T05:14:07.914069+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:14:07.914069+0000","last_undegraded":"2023-10-21T05:14:07.914069+0000","last_fullsized":"2023-10-21T05:14:07.914069+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7429,"log_dups_size":0,"ondisk_log_size":7429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1519","reported_seq":2513,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:04.333335+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:14:04.333335+0000","last_peered":"2023-10-21T05:14:04.333335+0000","last_clean":"2023-10-21T05:14:04.333335+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:14:04.333335+0000","last_undegraded":"2023-10-21T05:14:04.333335+0000","last_fullsized":"2023-10-21T05:14:04.333335+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1519,"log_dups_size":0,"ondisk_log_size":1519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1537","reported_seq":2652,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:04.333262+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:14:04.333262+0000","last_peered":"2023-10-21T05:14:04.333262+0000","last_clean":"2023-10-21T05:14:04.333262+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:14:04.333262+0000","last_undegraded":"2023-10-21T05:14:04.333262+0000","last_fullsized":"2023-10-21T05:14:04.333262+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1537,"log_dups_size":0,"ondisk_log_size":1537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'230451","reported_seq":462617,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:07.397845+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:14:07.397845+0000","last_peered":"2023-10-21T05:14:07.397845+0000","last_clean":"2023-10-21T05:14:07.397845+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:14:07.397845+0000","last_undegraded":"2023-10-21T05:14:07.397845+0000","last_fullsized":"2023-10-21T05:14:07.397845+0000","mapping_epoch":17,"log_start":"54'220440","ondisk_log_start":"54'220440","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":491,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 163s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5521518592,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":231990,"num_read_kb":901680,"num_write":230451,"num_write_kb":921804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1608","reported_seq":2610,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:02.797746+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:14:02.797746+0000","last_peered":"2023-10-21T05:14:02.797746+0000","last_clean":"2023-10-21T05:14:02.797746+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:14:02.797746+0000","last_undegraded":"2023-10-21T05:14:02.797746+0000","last_fullsized":"2023-10-21T05:14:02.797746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1608,"log_dups_size":0,"ondisk_log_size":1608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'215341","reported_seq":433358,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:10.845858+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:14:10.845858+0000","last_peered":"2023-10-21T05:14:10.845858+0000","last_clean":"2023-10-21T05:14:10.845858+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:14:10.845858+0000","last_undegraded":"2023-10-21T05:14:10.845858+0000","last_fullsized":"2023-10-21T05:14:10.845858+0000","mapping_epoch":17,"log_start":"53'205260","ondisk_log_start":"53'205260","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":502,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 156s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5189533696,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":217833,"num_read_kb":846252,"num_write":215341,"num_write_kb":861364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1003,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:02.804220+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:14:02.804220+0000","last_peered":"2023-10-21T05:14:02.804220+0000","last_clean":"2023-10-21T05:14:02.804220+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:14:02.804220+0000","last_undegraded":"2023-10-21T05:14:02.804220+0000","last_fullsized":"2023-10-21T05:14:02.804220+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'223808","reported_seq":450194,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:07.375249+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:14:07.375249+0000","last_peered":"2023-10-21T05:14:07.375249+0000","last_clean":"2023-10-21T05:14:07.375249+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:14:07.375249+0000","last_undegraded":"2023-10-21T05:14:07.375249+0000","last_fullsized":"2023-10-21T05:14:07.375249+0000","mapping_epoch":17,"log_start":"53'213709","ondisk_log_start":"53'213709","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":443,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 162s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5384028160,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":226206,"num_read_kb":879760,"num_write":223808,"num_write_kb":895232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3704,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:07.930559+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:14:07.930559+0000","last_peered":"2023-10-21T05:14:07.930559+0000","last_clean":"2023-10-21T05:14:07.930559+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:14:07.930559+0000","last_undegraded":"2023-10-21T05:14:07.930559+0000","last_fullsized":"2023-10-21T05:14:07.930559+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'218791","reported_seq":440722,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:10.843923+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:14:10.843923+0000","last_peered":"2023-10-21T05:14:10.843923+0000","last_clean":"2023-10-21T05:14:10.843923+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:14:10.843923+0000","last_undegraded":"2023-10-21T05:14:10.843923+0000","last_fullsized":"2023-10-21T05:14:10.843923+0000","mapping_epoch":17,"log_start":"53'208706","ondisk_log_start":"53'208706","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5259616275,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":221825,"num_read_kb":862328,"num_write":218791,"num_write_kb":875158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5913","reported_seq":9514,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:07.922699+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:14:07.922699+0000","last_peered":"2023-10-21T05:14:07.922699+0000","last_clean":"2023-10-21T05:14:07.922699+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:14:07.922699+0000","last_undegraded":"2023-10-21T05:14:07.922699+0000","last_fullsized":"2023-10-21T05:14:07.922699+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":5913,"log_dups_size":0,"ondisk_log_size":5913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":225,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:59.213787+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:59.213787+0000","last_peered":"2023-10-21T05:11:59.213787+0000","last_clean":"2023-10-21T05:11:59.213787+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:59.213787+0000","last_undegraded":"2023-10-21T05:11:59.213787+0000","last_fullsized":"2023-10-21T05:11:59.213787+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21354696723,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":897854,"num_read_kb":3490020,"num_write":888391,"num_write_kb":3553558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32046333952,"data_stored":32046333952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40276,"ondisk_log_size":40276,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":22578,"ondisk_log_size":22578,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739196,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11461168,"kb_used_data":10432232,"kb_used_omap":8,"kb_used_meta":1028919,"kb_avail":82910672,"statfs":{"total":96636764160,"available":84900528128,"internally_reserved":0,"allocated":10682605568,"data_stored":10682301037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1053613725},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.81799999999999995,"15min":0.871},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":4.4290000000000003,"5min":7.5579999999999998,"15min":22.969000000000001},"last":0.253},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.623,"15min":0.70299999999999996},"min":{"1min":0.216,"5min":0.20699999999999999,"15min":0.16500000000000001},"max":{"1min":0.441,"5min":5.1689999999999996,"15min":22.835000000000001},"last":4.7089999999999996}]},{"osd":1,"last update":"Sat Oct 21 05:13:27 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.77300000000000002,"15min":0.629},"min":{"1min":0.219,"5min":0.19,"15min":0.186},"max":{"1min":4.4740000000000002,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":1.0589999999999999,"15min":0.94199999999999995},"min":{"1min":0.26000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":3.7909999999999999,"5min":21.234999999999999,"15min":21.234999999999999},"last":0.28100000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739195,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11517268,"kb_used_data":10432908,"kb_used_omap":15,"kb_used_meta":1084336,"kb_avail":82854572,"statfs":{"total":96636764160,"available":84843081728,"internally_reserved":0,"allocated":10683297792,"data_stored":10682879996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1110360454},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,8,10,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:20 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.69399999999999995,"15min":0.76600000000000001},"min":{"1min":0.16300000000000001,"5min":0.13,"15min":0.13},"max":{"1min":4.4640000000000004,"5min":5.657,"15min":11.225},"last":0.214},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.48099999999999998,"15min":0.51500000000000001},"min":{"1min":0.19700000000000001,"5min":0.185,"15min":0.18099999999999999},"max":{"1min":2.1560000000000001,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.23999999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:13:20 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.86399999999999999,"15min":0.77400000000000002},"min":{"1min":0.23799999999999999,"5min":0.185,"15min":0.185},"max":{"1min":2.1389999999999998,"5min":6.2060000000000004,"15min":8.5389999999999997},"last":0.442},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.86299999999999999,"15min":0.79400000000000004},"min":{"1min":0.28799999999999998,"5min":0.224,"15min":0.20599999999999999},"max":{"1min":2.181,"5min":6.1669999999999998,"15min":9.1950000000000003},"last":0.47099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739196,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11446876,"kb_used_data":10432852,"kb_used_omap":12,"kb_used_meta":1014003,"kb_avail":82924964,"statfs":{"total":96636764160,"available":84915163136,"internally_reserved":0,"allocated":10683240448,"data_stored":10682875451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1038339497},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,1,6,5,17,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:26 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.59299999999999997,"15min":0.56899999999999995},"min":{"1min":0.188,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":5.5709999999999997,"5min":6.5759999999999996,"15min":11.773},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.61799999999999999,"15min":0.64800000000000002},"min":{"1min":0.26200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":5.6100000000000003,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.504}]},{"osd":2,"last update":"Sat Oct 21 05:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":0.88,"15min":0.66400000000000003},"min":{"1min":0.33500000000000002,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":5.6529999999999996,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.026,"15min":0.85599999999999998},"min":{"1min":0.21199999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":3.8889999999999998,"5min":9.516,"15min":10.074999999999999},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10682097664,"data_stored":10682097664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10682122240,"data_stored":10682122240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10682114048,"data_stored":10682114048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:14:12.880 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:12.881 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:12.881 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:12.882 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:12.883 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:32.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:14:33.560 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:14:33.562 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:14:33.590 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2133,"stamp":"2023-10-21T05:14:31.669262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21355726691,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":907085,"num_read_kb":3508655,"num_write":892493,"num_write_kb":3570463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63043,"ondisk_log_size":63043,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":34602284,"kb_used_data":31298632,"kb_used_omap":36,"kb_used_meta":3303579,"kb_avail":248513236,"statfs":{"total":289910292480,"available":254477553664,"internally_reserved":0,"allocated":32049799168,"data_stored":32048711844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":3382865356},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,2,2,8,16,15,15,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":270336,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2126,"num_read_kb":8416,"num_write":2295,"num_write_kb":9180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002202"},"pg_stats":[{"pgid":"2.7","version":"54'4512","reported_seq":7271,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:30.574337+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:14:30.574337+0000","last_peered":"2023-10-21T05:14:30.574337+0000","last_clean":"2023-10-21T05:14:30.574337+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:14:30.574337+0000","last_undegraded":"2023-10-21T05:14:30.574337+0000","last_fullsized":"2023-10-21T05:14:30.574337+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4512,"log_dups_size":0,"ondisk_log_size":4512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7479","reported_seq":12231,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:30.585204+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:14:30.585204+0000","last_peered":"2023-10-21T05:14:30.585204+0000","last_clean":"2023-10-21T05:14:30.585204+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:14:30.585204+0000","last_undegraded":"2023-10-21T05:14:30.585204+0000","last_fullsized":"2023-10-21T05:14:30.585204+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7479,"log_dups_size":0,"ondisk_log_size":7479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1529","reported_seq":2527,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:29.932206+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:14:29.932206+0000","last_peered":"2023-10-21T05:14:29.932206+0000","last_clean":"2023-10-21T05:14:29.932206+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:14:29.932206+0000","last_undegraded":"2023-10-21T05:14:29.932206+0000","last_fullsized":"2023-10-21T05:14:29.932206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1529,"log_dups_size":0,"ondisk_log_size":1529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1547","reported_seq":2666,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:29.937930+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:14:29.937930+0000","last_peered":"2023-10-21T05:14:29.937930+0000","last_clean":"2023-10-21T05:14:29.937930+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:14:29.937930+0000","last_undegraded":"2023-10-21T05:14:29.937930+0000","last_fullsized":"2023-10-21T05:14:29.937930+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1547,"log_dups_size":0,"ondisk_log_size":1547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'231420","reported_seq":464514,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:27.611022+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:14:27.611022+0000","last_peered":"2023-10-21T05:14:27.611022+0000","last_clean":"2023-10-21T05:14:27.611022+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:14:27.611022+0000","last_undegraded":"2023-10-21T05:14:27.611022+0000","last_fullsized":"2023-10-21T05:14:27.611022+0000","mapping_epoch":17,"log_start":"54'221351","ondisk_log_start":"54'221351","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":537,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 183s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5521653760,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":232906,"num_read_kb":905316,"num_write":231420,"num_write_kb":925680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1614","reported_seq":2621,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:23.305941+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:14:23.305941+0000","last_peered":"2023-10-21T05:14:23.305941+0000","last_clean":"2023-10-21T05:14:23.305941+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:14:23.305941+0000","last_undegraded":"2023-10-21T05:14:23.305941+0000","last_fullsized":"2023-10-21T05:14:23.305941+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1614,"log_dups_size":0,"ondisk_log_size":1614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'216215","reported_seq":435056,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:31.074056+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:14:31.074056+0000","last_peered":"2023-10-21T05:14:31.074056+0000","last_clean":"2023-10-21T05:14:31.074056+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:14:31.074056+0000","last_undegraded":"2023-10-21T05:14:31.074056+0000","last_fullsized":"2023-10-21T05:14:31.074056+0000","mapping_epoch":17,"log_start":"54'206175","ondisk_log_start":"54'206175","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":588,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 176s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5189652480,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":218640,"num_read_kb":849424,"num_write":216215,"num_write_kb":864860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1008,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:23.286501+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:14:23.286501+0000","last_peered":"2023-10-21T05:14:23.286501+0000","last_clean":"2023-10-21T05:14:23.286501+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:14:23.286501+0000","last_undegraded":"2023-10-21T05:14:23.286501+0000","last_fullsized":"2023-10-21T05:14:23.286501+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'224735","reported_seq":451986,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:27.600904+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:14:27.600904+0000","last_peered":"2023-10-21T05:14:27.600904+0000","last_clean":"2023-10-21T05:14:27.600904+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:14:27.600904+0000","last_undegraded":"2023-10-21T05:14:27.600904+0000","last_fullsized":"2023-10-21T05:14:27.600904+0000","mapping_epoch":17,"log_start":"54'214638","ondisk_log_start":"54'214638","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":547,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 182s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5384105984,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":227063,"num_read_kb":883160,"num_write":224735,"num_write_kb":898940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3720,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:28.411936+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:14:28.411936+0000","last_peered":"2023-10-21T05:14:28.411936+0000","last_clean":"2023-10-21T05:14:28.411936+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:14:28.411936+0000","last_undegraded":"2023-10-21T05:14:28.411936+0000","last_fullsized":"2023-10-21T05:14:28.411936+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'219723","reported_seq":442541,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:31.110550+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:14:31.110550+0000","last_peered":"2023-10-21T05:14:31.110550+0000","last_clean":"2023-10-21T05:14:31.110550+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:14:31.110550+0000","last_undegraded":"2023-10-21T05:14:31.110550+0000","last_fullsized":"2023-10-21T05:14:31.110550+0000","mapping_epoch":17,"log_start":"54'209709","ondisk_log_start":"54'209709","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5259722771,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":222710,"num_read_kb":865848,"num_write":219723,"num_write_kb":878886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5953","reported_seq":9570,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:30.711419+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:14:30.711419+0000","last_peered":"2023-10-21T05:14:30.711419+0000","last_clean":"2023-10-21T05:14:30.711419+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:14:30.711419+0000","last_undegraded":"2023-10-21T05:14:30.711419+0000","last_fullsized":"2023-10-21T05:14:30.711419+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":5953,"log_dups_size":0,"ondisk_log_size":5953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":225,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:59.213787+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:59.213787+0000","last_peered":"2023-10-21T05:11:59.213787+0000","last_clean":"2023-10-21T05:11:59.213787+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:59.213787+0000","last_undegraded":"2023-10-21T05:11:59.213787+0000","last_fullsized":"2023-10-21T05:11:59.213787+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21355134995,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":901319,"num_read_kb":3503748,"num_write":892093,"num_write_kb":3568366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32046895104,"data_stored":32046895104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40220,"ondisk_log_size":40220,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":22724,"ondisk_log_size":22724,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739200,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11533560,"kb_used_data":10432428,"kb_used_omap":8,"kb_used_meta":1101111,"kb_avail":82838280,"statfs":{"total":96636764160,"available":84826398720,"internally_reserved":0,"allocated":10682806272,"data_stored":10682501741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1127538333},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.81799999999999995,"15min":0.871},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":4.4290000000000003,"5min":7.5579999999999998,"15min":22.969000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.623,"15min":0.70299999999999996},"min":{"1min":0.216,"5min":0.20699999999999999,"15min":0.16500000000000001},"max":{"1min":0.441,"5min":5.1689999999999996,"15min":22.835000000000001},"last":1.014}]},{"osd":1,"last update":"Sat Oct 21 05:14:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.81899999999999995,"15min":0.60699999999999998},"min":{"1min":0.192,"5min":0.19,"15min":0.186},"max":{"1min":1.8029999999999999,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":1.004,"15min":0.90200000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.821,"5min":21.234999999999999,"15min":21.234999999999999},"last":1.048}]}]},{"osd":1,"up_from":8,"seq":34359739199,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11553600,"kb_used_data":10433136,"kb_used_omap":15,"kb_used_meta":1120432,"kb_avail":82818240,"statfs":{"total":96636764160,"available":84805877760,"internally_reserved":0,"allocated":10683531264,"data_stored":10683113468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1147322758},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,2,0,3,2,7,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:21 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":0.76100000000000001,"15min":0.79300000000000004},"min":{"1min":0.17000000000000001,"5min":0.13300000000000001,"15min":0.13},"max":{"1min":14.682,"5min":14.682,"15min":14.682},"last":0.189},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54000000000000004,"15min":0.497},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.9140000000000001,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.23999999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:14:21 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.76900000000000002,"15min":0.76400000000000001},"min":{"1min":0.23899999999999999,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":2.8359999999999999,"5min":5.7039999999999997,"15min":8.5389999999999997},"last":2.4140000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.72799999999999998,"15min":0.78400000000000003},"min":{"1min":0.27800000000000002,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":2.871,"5min":5.7400000000000002,"15min":9.1950000000000003},"last":2.4460000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739200,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11515124,"kb_used_data":10433068,"kb_used_omap":12,"kb_used_meta":1082035,"kb_avail":82856716,"statfs":{"total":96636764160,"available":84845277184,"internally_reserved":0,"allocated":10683461632,"data_stored":10683096635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1108004265},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,11,7,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:26 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.59299999999999997,"15min":0.56899999999999995},"min":{"1min":0.188,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":5.5709999999999997,"5min":6.5759999999999996,"15min":11.773},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.61799999999999999,"15min":0.64800000000000002},"min":{"1min":0.26200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":5.6100000000000003,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.45800000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":0.88,"15min":0.66400000000000003},"min":{"1min":0.33500000000000002,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":5.6529999999999996,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.026,"15min":0.85599999999999998},"min":{"1min":0.21199999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":3.8889999999999998,"5min":9.516,"15min":10.074999999999999},"last":0.43099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10682298368,"data_stored":10682298368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10682302464,"data_stored":10682302464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10682294272,"data_stored":10682294272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:14:33.592 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:33.593 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:33.593 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:33.594 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:33.594 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:49.917 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:14:49.912+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:14:53.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:14:54.364 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:14:54.365 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:14:54.428 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2144,"stamp":"2023-10-21T05:14:53.673195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21356103523,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":911045,"num_read_kb":3524315,"num_write":896684,"num_write_kb":3587227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63006,"ondisk_log_size":63006,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":34652744,"kb_used_data":31299092,"kb_used_omap":36,"kb_used_meta":3353563,"kb_avail":248462776,"statfs":{"total":289910292480,"available":254425882624,"internally_reserved":0,"allocated":32050270208,"data_stored":32049182884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":3434048972},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,6,7,11,10,23,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":200704,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2273,"num_read_kb":8972,"num_write":2320,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002139"},"pg_stats":[{"pgid":"2.7","version":"54'4522","reported_seq":7293,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:50.426993+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:14:50.426993+0000","last_peered":"2023-10-21T05:14:50.426993+0000","last_clean":"2023-10-21T05:14:50.426993+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:14:50.426993+0000","last_undegraded":"2023-10-21T05:14:50.426993+0000","last_fullsized":"2023-10-21T05:14:50.426993+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7494","reported_seq":12266,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:50.439139+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:14:50.439139+0000","last_peered":"2023-10-21T05:14:50.439139+0000","last_clean":"2023-10-21T05:14:50.439139+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:14:50.439139+0000","last_undegraded":"2023-10-21T05:14:50.439139+0000","last_fullsized":"2023-10-21T05:14:50.439139+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7494,"log_dups_size":0,"ondisk_log_size":7494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1532","reported_seq":2535,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:50.420602+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:14:50.420602+0000","last_peered":"2023-10-21T05:14:50.420602+0000","last_clean":"2023-10-21T05:14:50.420602+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:14:50.420602+0000","last_undegraded":"2023-10-21T05:14:50.420602+0000","last_fullsized":"2023-10-21T05:14:50.420602+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1532,"log_dups_size":0,"ondisk_log_size":1532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1550","reported_seq":2674,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:49.912060+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:14:49.912060+0000","last_peered":"2023-10-21T05:14:49.912060+0000","last_clean":"2023-10-21T05:14:49.912060+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:14:49.912060+0000","last_undegraded":"2023-10-21T05:14:49.912060+0000","last_fullsized":"2023-10-21T05:14:49.912060+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1550,"log_dups_size":0,"ondisk_log_size":1550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'232609","reported_seq":466849,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:52.940031+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:14:52.940031+0000","last_peered":"2023-10-21T05:14:52.940031+0000","last_clean":"2023-10-21T05:14:52.940031+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:14:52.940031+0000","last_undegraded":"2023-10-21T05:14:52.940031+0000","last_fullsized":"2023-10-21T05:14:52.940031+0000","mapping_epoch":17,"log_start":"54'222583","ondisk_log_start":"54'222583","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":593,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 208s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5521776640,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":234043,"num_read_kb":909816,"num_write":232609,"num_write_kb":930436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1619","reported_seq":2631,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:47.436798+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:14:47.436798+0000","last_peered":"2023-10-21T05:14:47.436798+0000","last_clean":"2023-10-21T05:14:47.436798+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:14:47.436798+0000","last_undegraded":"2023-10-21T05:14:47.436798+0000","last_fullsized":"2023-10-21T05:14:47.436798+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1619,"log_dups_size":0,"ondisk_log_size":1619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'217103","reported_seq":436835,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:51.341150+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:14:51.341150+0000","last_peered":"2023-10-21T05:14:51.341150+0000","last_clean":"2023-10-21T05:14:51.341150+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:14:51.341150+0000","last_undegraded":"2023-10-21T05:14:51.341150+0000","last_fullsized":"2023-10-21T05:14:51.341150+0000","mapping_epoch":17,"log_start":"54'207091","ondisk_log_start":"54'207091","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":672,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 196s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5189734400,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":219520,"num_read_kb":852900,"num_write":217103,"num_write_kb":868412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1012,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:43.754315+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:14:43.754315+0000","last_peered":"2023-10-21T05:14:43.754315+0000","last_clean":"2023-10-21T05:14:43.754315+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:14:43.754315+0000","last_undegraded":"2023-10-21T05:14:43.754315+0000","last_fullsized":"2023-10-21T05:14:43.754315+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'225924","reported_seq":454286,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:52.914725+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:14:52.914725+0000","last_peered":"2023-10-21T05:14:52.914725+0000","last_clean":"2023-10-21T05:14:52.914725+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:14:52.914725+0000","last_undegraded":"2023-10-21T05:14:52.914725+0000","last_fullsized":"2023-10-21T05:14:52.914725+0000","mapping_epoch":17,"log_start":"54'215858","ondisk_log_start":"54'215858","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":651,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 207s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5384228864,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":228155,"num_read_kb":887488,"num_write":225924,"num_write_kb":903696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3736,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:48.875480+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:14:48.875480+0000","last_peered":"2023-10-21T05:14:48.875480+0000","last_clean":"2023-10-21T05:14:48.875480+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:14:48.875480+0000","last_undegraded":"2023-10-21T05:14:48.875480+0000","last_fullsized":"2023-10-21T05:14:48.875480+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'220648","reported_seq":444318,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:51.396776+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:14:51.396776+0000","last_peered":"2023-10-21T05:14:51.396776+0000","last_clean":"2023-10-21T05:14:51.396776+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:14:51.396776+0000","last_undegraded":"2023-10-21T05:14:51.396776+0000","last_fullsized":"2023-10-21T05:14:51.396776+0000","mapping_epoch":17,"log_start":"54'210617","ondisk_log_start":"54'210617","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5259771923,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":223561,"num_read_kb":869204,"num_write":220648,"num_write_kb":882586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5965","reported_seq":9599,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:50.617030+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:14:50.617030+0000","last_peered":"2023-10-21T05:14:50.617030+0000","last_clean":"2023-10-21T05:14:50.617030+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:14:50.617030+0000","last_undegraded":"2023-10-21T05:14:50.617030+0000","last_fullsized":"2023-10-21T05:14:50.617030+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":5965,"log_dups_size":0,"ondisk_log_size":5965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":225,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:59.213787+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:59.213787+0000","last_peered":"2023-10-21T05:11:59.213787+0000","last_clean":"2023-10-21T05:11:59.213787+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:59.213787+0000","last_undegraded":"2023-10-21T05:11:59.213787+0000","last_fullsized":"2023-10-21T05:11:59.213787+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21355511827,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":905279,"num_read_kb":3519408,"num_write":896284,"num_write_kb":3585130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32047423488,"data_stored":32047423488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40135,"ondisk_log_size":40135,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":22772,"ondisk_log_size":22772,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739204,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11547608,"kb_used_data":10432592,"kb_used_omap":8,"kb_used_meta":1114999,"kb_avail":82824232,"statfs":{"total":96636764160,"available":84812013568,"internally_reserved":0,"allocated":10682974208,"data_stored":10682669677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1141759645},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.72399999999999998,"15min":0.86499999999999999},"min":{"1min":0.23300000000000001,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":0.94399999999999995,"5min":5.7169999999999996,"15min":22.969000000000001},"last":0.216},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.67500000000000004,"15min":0.71799999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.16500000000000001},"max":{"1min":4.7089999999999996,"5min":5.1689999999999996,"15min":22.835000000000001},"last":0.30299999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:14:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.81899999999999995,"15min":0.60699999999999998},"min":{"1min":0.192,"5min":0.19,"15min":0.186},"max":{"1min":1.8029999999999999,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.245},{"interface":"front","average":{"1min":0.51700000000000002,"5min":1.004,"15min":0.90200000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.821,"5min":21.234999999999999,"15min":21.234999999999999},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739203,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11567612,"kb_used_data":10433260,"kb_used_omap":15,"kb_used_meta":1134320,"kb_avail":82804228,"statfs":{"total":96636764160,"available":84791529472,"internally_reserved":0,"allocated":10683658240,"data_stored":10683240444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1161544070},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1,4,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:21 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":0.76100000000000001,"15min":0.79300000000000004},"min":{"1min":0.17000000000000001,"5min":0.13300000000000001,"15min":0.13},"max":{"1min":14.682,"5min":14.682,"15min":14.682},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54000000000000004,"15min":0.497},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.9140000000000001,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.23300000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:14:21 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.76900000000000002,"15min":0.76400000000000001},"min":{"1min":0.23899999999999999,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":2.8359999999999999,"5min":5.7039999999999997,"15min":8.5389999999999997},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.72799999999999998,"15min":0.78400000000000003},"min":{"1min":0.27800000000000002,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":2.871,"5min":5.7400000000000002,"15min":9.1950000000000003},"last":0.70299999999999996}]}]},{"osd":0,"up_from":8,"seq":34359739205,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11537524,"kb_used_data":10433240,"kb_used_omap":12,"kb_used_meta":1104243,"kb_avail":82834316,"statfs":{"total":96636764160,"available":84822339584,"internally_reserved":0,"allocated":10683637760,"data_stored":10683272763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1130745257},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,4,4,5,19,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.59099999999999997,"15min":0.54400000000000004},"min":{"1min":0.20300000000000001,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":6.5759999999999996,"15min":11.773},"last":0.252},{"interface":"front","average":{"1min":0.377,"5min":0.60899999999999999,"15min":0.60299999999999998},"min":{"1min":0.247,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":0.58099999999999996,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Oct 21 05:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.84699999999999998,"15min":0.66900000000000004},"min":{"1min":0.245,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":3.9620000000000002,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":1.0009999999999999,"15min":0.80100000000000005},"min":{"1min":0.23799999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":4.4409999999999998,"5min":9.516,"15min":10.074999999999999},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10682511360,"data_stored":10682511360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10682433536,"data_stored":10682433536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10682478592,"data_stored":10682478592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:14:54.429 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:54.430 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:54.430 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:54.431 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:54.431 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:15:14.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:15:15.255 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:15:15.256 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:15:15.292 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2154,"stamp":"2023-10-21T05:15:13.679104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21356742499,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":914716,"num_read_kb":3538851,"num_write":900390,"num_write_kb":3602051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63102,"ondisk_log_size":63102,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":34689484,"kb_used_data":31299828,"kb_used_omap":36,"kb_used_meta":3389595,"kb_avail":248426036,"statfs":{"total":289910292480,"available":254388260864,"internally_reserved":0,"allocated":32051023872,"data_stored":32049936548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":3470945740},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,6,3,16,16,24,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":434176,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2259,"num_read_kb":8932,"num_write":2358,"num_write_kb":9432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004422"},"pg_stats":[{"pgid":"2.7","version":"54'4528","reported_seq":7312,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:09.354372+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:15:09.354372+0000","last_peered":"2023-10-21T05:15:09.354372+0000","last_clean":"2023-10-21T05:15:09.354372+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:15:09.354372+0000","last_undegraded":"2023-10-21T05:15:09.354372+0000","last_fullsized":"2023-10-21T05:15:09.354372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7504","reported_seq":12296,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:09.354787+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:15:09.354787+0000","last_peered":"2023-10-21T05:15:09.354787+0000","last_clean":"2023-10-21T05:15:09.354787+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:15:09.354787+0000","last_undegraded":"2023-10-21T05:15:09.354787+0000","last_fullsized":"2023-10-21T05:15:09.354787+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7504,"log_dups_size":0,"ondisk_log_size":7504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1534","reported_seq":2541,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:07.422791+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:15:07.422791+0000","last_peered":"2023-10-21T05:15:07.422791+0000","last_clean":"2023-10-21T05:15:07.422791+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:15:07.422791+0000","last_undegraded":"2023-10-21T05:15:07.422791+0000","last_fullsized":"2023-10-21T05:15:07.422791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1534,"log_dups_size":0,"ondisk_log_size":1534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1552","reported_seq":2680,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:07.422812+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:15:07.422812+0000","last_peered":"2023-10-21T05:15:07.422812+0000","last_clean":"2023-10-21T05:15:07.422812+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:15:07.422812+0000","last_undegraded":"2023-10-21T05:15:07.422812+0000","last_fullsized":"2023-10-21T05:15:07.422812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1552,"log_dups_size":0,"ondisk_log_size":1552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'233568","reported_seq":468747,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:13.207002+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:15:13.207002+0000","last_peered":"2023-10-21T05:15:13.207002+0000","last_clean":"2023-10-21T05:15:13.207002+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:15:13.207002+0000","last_undegraded":"2023-10-21T05:15:13.207002+0000","last_fullsized":"2023-10-21T05:15:13.207002+0000","mapping_epoch":17,"log_start":"54'223498","ondisk_log_start":"54'223498","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":631,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 229s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5521924096,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":234973,"num_read_kb":913504,"num_write":233568,"num_write_kb":934272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1626","reported_seq":2642,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:12.787634+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:15:12.787634+0000","last_peered":"2023-10-21T05:15:12.787634+0000","last_clean":"2023-10-21T05:15:12.787634+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:15:12.787634+0000","last_undegraded":"2023-10-21T05:15:12.787634+0000","last_fullsized":"2023-10-21T05:15:12.787634+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1626,"log_dups_size":0,"ondisk_log_size":1626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'218002","reported_seq":438671,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:11.663592+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:15:11.663592+0000","last_peered":"2023-10-21T05:15:11.663592+0000","last_clean":"2023-10-21T05:15:11.663592+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:15:11.663592+0000","last_undegraded":"2023-10-21T05:15:11.663592+0000","last_fullsized":"2023-10-21T05:15:11.663592+0000","mapping_epoch":17,"log_start":"54'208000","ondisk_log_start":"54'208000","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":705,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 216s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5189881856,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":220449,"num_read_kb":856580,"num_write":218002,"num_write_kb":872008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1015,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:59.123533+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:14:59.123533+0000","last_peered":"2023-10-21T05:14:59.123533+0000","last_clean":"2023-10-21T05:14:59.123533+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:14:59.123533+0000","last_undegraded":"2023-10-21T05:14:59.123533+0000","last_fullsized":"2023-10-21T05:14:59.123533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'226881","reported_seq":456142,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:13.203473+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:15:13.203473+0000","last_peered":"2023-10-21T05:15:13.203473+0000","last_clean":"2023-10-21T05:15:13.203473+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:15:13.203473+0000","last_undegraded":"2023-10-21T05:15:13.203473+0000","last_fullsized":"2023-10-21T05:15:13.203473+0000","mapping_epoch":17,"log_start":"54'216779","ondisk_log_start":"54'216779","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":708,"log_size":10102,"log_dups_size":0,"ondisk_log_size":10102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 228s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5384388608,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":229044,"num_read_kb":891016,"num_write":226881,"num_write_kb":907524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3748,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:04.242183+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:15:04.242183+0000","last_peered":"2023-10-21T05:15:04.242183+0000","last_clean":"2023-10-21T05:15:04.242183+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:15:04.242183+0000","last_undegraded":"2023-10-21T05:15:04.242183+0000","last_fullsized":"2023-10-21T05:15:04.242183+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'221539","reported_seq":446130,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:11.708643+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:15:11.708643+0000","last_peered":"2023-10-21T05:15:11.708643+0000","last_clean":"2023-10-21T05:15:11.708643+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:15:11.708643+0000","last_undegraded":"2023-10-21T05:15:11.708643+0000","last_fullsized":"2023-10-21T05:15:11.708643+0000","mapping_epoch":17,"log_start":"54'211517","ondisk_log_start":"54'211517","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5259956243,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":224484,"num_read_kb":872844,"num_write":221539,"num_write_kb":886150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'5973","reported_seq":9623,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:09.360908+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:15:09.360908+0000","last_peered":"2023-10-21T05:15:09.360908+0000","last_clean":"2023-10-21T05:15:09.360908+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:15:09.360908+0000","last_undegraded":"2023-10-21T05:15:09.360908+0000","last_fullsized":"2023-10-21T05:15:09.360908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":5973,"log_dups_size":0,"ondisk_log_size":5973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":225,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:59.213787+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:59.213787+0000","last_peered":"2023-10-21T05:11:59.213787+0000","last_clean":"2023-10-21T05:11:59.213787+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:59.213787+0000","last_undegraded":"2023-10-21T05:11:59.213787+0000","last_fullsized":"2023-10-21T05:11:59.213787+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21356150803,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":908950,"num_read_kb":3533944,"num_write":899990,"num_write_kb":3599954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32048177152,"data_stored":32048177152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40196,"ondisk_log_size":40196,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":22807,"ondisk_log_size":22807,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739208,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11547768,"kb_used_data":10432752,"kb_used_omap":8,"kb_used_meta":1114999,"kb_avail":82824072,"statfs":{"total":96636764160,"available":84811849728,"internally_reserved":0,"allocated":10683138048,"data_stored":10682833517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1141759645},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.72399999999999998,"15min":0.86499999999999999},"min":{"1min":0.23300000000000001,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":0.94399999999999995,"5min":5.7169999999999996,"15min":22.969000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.67500000000000004,"15min":0.71799999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.16500000000000001},"max":{"1min":4.7089999999999996,"5min":5.1689999999999996,"15min":22.835000000000001},"last":0.309}]},{"osd":1,"last update":"Sat Oct 21 05:14:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.81899999999999995,"15min":0.60699999999999998},"min":{"1min":0.192,"5min":0.19,"15min":0.186},"max":{"1min":1.8029999999999999,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":1.004,"15min":0.90200000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.821,"5min":21.234999999999999,"15min":21.234999999999999},"last":0.441}]}]},{"osd":1,"up_from":8,"seq":34359739207,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11567884,"kb_used_data":10433540,"kb_used_omap":15,"kb_used_meta":1134320,"kb_avail":82803956,"statfs":{"total":96636764160,"available":84791250944,"internally_reserved":0,"allocated":10683944960,"data_stored":10683527164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1161544070},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,10,10,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:21 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":0.76100000000000001,"15min":0.79300000000000004},"min":{"1min":0.17000000000000001,"5min":0.13300000000000001,"15min":0.13},"max":{"1min":14.682,"5min":14.682,"15min":14.682},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54000000000000004,"15min":0.497},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.9140000000000001,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.502}]},{"osd":2,"last update":"Sat Oct 21 05:14:21 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.76900000000000002,"15min":0.76400000000000001},"min":{"1min":0.23899999999999999,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":2.8359999999999999,"5min":5.7039999999999997,"15min":8.5389999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.72799999999999998,"15min":0.78400000000000003},"min":{"1min":0.27800000000000002,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":2.871,"5min":5.7400000000000002,"15min":9.1950000000000003},"last":0.56499999999999995}]}]},{"osd":0,"up_from":8,"seq":34359739209,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11573832,"kb_used_data":10433536,"kb_used_omap":12,"kb_used_meta":1140275,"kb_avail":82798008,"statfs":{"total":96636764160,"available":84785160192,"internally_reserved":0,"allocated":10683940864,"data_stored":10683575867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1167642025},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,0,5,5,17,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.59099999999999997,"15min":0.54400000000000004},"min":{"1min":0.20300000000000001,"5min":0.17699999999999999,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":6.5759999999999996,"15min":11.773},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.60899999999999999,"15min":0.60299999999999998},"min":{"1min":0.247,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":0.58099999999999996,"5min":6.6429999999999998,"15min":11.808999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.84699999999999998,"15min":0.66900000000000004},"min":{"1min":0.245,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":3.9620000000000002,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":1.0009999999999999,"15min":0.80100000000000005},"min":{"1min":0.23799999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":4.4409999999999998,"5min":9.516,"15min":10.074999999999999},"last":0.48299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10682765312,"data_stored":10682765312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10682769408,"data_stored":10682769408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10682642432,"data_stored":10682642432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:15:15.294 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:15.294 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:15.294 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:15.295 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:15.295 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:15:29.460 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:15:29.451+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:15:29.468 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:15:29.451+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:15:35.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:15:35.934 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:15:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:15:36.009 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2165,"stamp":"2023-10-21T05:15:35.684611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21357483875,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":918473,"num_read_kb":3553715,"num_write":904020,"num_write_kb":3616571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63212,"ondisk_log_size":63212,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":34803064,"kb_used_data":31301080,"kb_used_omap":36,"kb_used_meta":3501915,"kb_avail":248312456,"statfs":{"total":289910292480,"available":254271954944,"internally_reserved":0,"allocated":32052305920,"data_stored":32051218596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":3585961420},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,4,7,2,6,12,23,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":421888,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1913,"num_read_kb":7544,"num_write":1810,"num_write_kb":7240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003660"},"pg_stats":[{"pgid":"2.7","version":"54'4556","reported_seq":7352,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:29.878521+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:15:29.878521+0000","last_peered":"2023-10-21T05:15:29.878521+0000","last_clean":"2023-10-21T05:15:29.878521+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:15:29.878521+0000","last_undegraded":"2023-10-21T05:15:29.878521+0000","last_fullsized":"2023-10-21T05:15:29.878521+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7549","reported_seq":12362,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:29.878951+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:15:29.878951+0000","last_peered":"2023-10-21T05:15:29.878951+0000","last_clean":"2023-10-21T05:15:29.878951+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:15:29.878951+0000","last_undegraded":"2023-10-21T05:15:29.878951+0000","last_fullsized":"2023-10-21T05:15:29.878951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7549,"log_dups_size":0,"ondisk_log_size":7549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1544","reported_seq":2555,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:29.445530+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:15:29.445530+0000","last_peered":"2023-10-21T05:15:29.445530+0000","last_clean":"2023-10-21T05:15:29.445530+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:15:29.445530+0000","last_undegraded":"2023-10-21T05:15:29.445530+0000","last_fullsized":"2023-10-21T05:15:29.445530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1544,"log_dups_size":0,"ondisk_log_size":1544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1561","reported_seq":2693,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:29.445529+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:15:29.445529+0000","last_peered":"2023-10-21T05:15:29.445529+0000","last_clean":"2023-10-21T05:15:29.445529+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:15:29.445529+0000","last_undegraded":"2023-10-21T05:15:29.445529+0000","last_fullsized":"2023-10-21T05:15:29.445529+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1561,"log_dups_size":0,"ondisk_log_size":1561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'234536","reported_seq":470711,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:33.431616+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:15:33.431616+0000","last_peered":"2023-10-21T05:15:33.431616+0000","last_clean":"2023-10-21T05:15:33.431616+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:15:33.431616+0000","last_undegraded":"2023-10-21T05:15:33.431616+0000","last_fullsized":"2023-10-21T05:15:33.431616+0000","mapping_epoch":17,"log_start":"54'224515","ondisk_log_start":"54'224515","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":703,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 249s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5522120704,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":235960,"num_read_kb":917416,"num_write":234536,"num_write_kb":938144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1631","reported_seq":2651,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:29.837601+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:15:29.837601+0000","last_peered":"2023-10-21T05:15:29.837601+0000","last_clean":"2023-10-21T05:15:29.837601+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:15:29.837601+0000","last_undegraded":"2023-10-21T05:15:29.837601+0000","last_fullsized":"2023-10-21T05:15:29.837601+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1631,"log_dups_size":0,"ondisk_log_size":1631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'218873","reported_seq":440452,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:31.935048+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:15:31.935048+0000","last_peered":"2023-10-21T05:15:31.935048+0000","last_clean":"2023-10-21T05:15:31.935048+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:15:31.935048+0000","last_undegraded":"2023-10-21T05:15:31.935048+0000","last_fullsized":"2023-10-21T05:15:31.935048+0000","mapping_epoch":17,"log_start":"54'208815","ondisk_log_start":"54'208815","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":789,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 237s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190021120,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":221346,"num_read_kb":860124,"num_write":218873,"num_write_kb":875492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1020,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:24.732218+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:15:24.732218+0000","last_peered":"2023-10-21T05:15:24.732218+0000","last_clean":"2023-10-21T05:15:24.732218+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:15:24.732218+0000","last_undegraded":"2023-10-21T05:15:24.732218+0000","last_fullsized":"2023-10-21T05:15:24.732218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'227775","reported_seq":457958,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:33.365536+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:15:33.365536+0000","last_peered":"2023-10-21T05:15:33.365536+0000","last_clean":"2023-10-21T05:15:33.365536+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:15:33.365536+0000","last_undegraded":"2023-10-21T05:15:33.365536+0000","last_fullsized":"2023-10-21T05:15:33.365536+0000","mapping_epoch":17,"log_start":"54'217691","ondisk_log_start":"54'217691","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":787,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 248s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5384708096,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":229957,"num_read_kb":894628,"num_write":227775,"num_write_kb":911100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3768,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:29.840219+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:15:29.840219+0000","last_peered":"2023-10-21T05:15:29.840219+0000","last_clean":"2023-10-21T05:15:29.840219+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:15:29.840219+0000","last_undegraded":"2023-10-21T05:15:29.840219+0000","last_fullsized":"2023-10-21T05:15:29.840219+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'222436","reported_seq":447987,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:31.954698+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:15:31.954698+0000","last_peered":"2023-10-21T05:15:31.954698+0000","last_clean":"2023-10-21T05:15:31.954698+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:15:31.954698+0000","last_undegraded":"2023-10-21T05:15:31.954698+0000","last_fullsized":"2023-10-21T05:15:31.954698+0000","mapping_epoch":17,"log_start":"54'212426","ondisk_log_start":"54'212426","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5260042259,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":225444,"num_read_kb":876640,"num_write":222436,"num_write_kb":889738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6009","reported_seq":9675,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:30.043677+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:15:30.043677+0000","last_peered":"2023-10-21T05:15:30.043677+0000","last_clean":"2023-10-21T05:15:30.043677+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:15:30.043677+0000","last_undegraded":"2023-10-21T05:15:30.043677+0000","last_fullsized":"2023-10-21T05:15:30.043677+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":225,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:59.213787+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:59.213787+0000","last_peered":"2023-10-21T05:11:59.213787+0000","last_clean":"2023-10-21T05:11:59.213787+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:59.213787+0000","last_undegraded":"2023-10-21T05:11:59.213787+0000","last_fullsized":"2023-10-21T05:11:59.213787+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21356892179,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":912707,"num_read_kb":3548808,"num_write":903620,"num_write_kb":3614474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32049430528,"data_stored":32049430528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40173,"ondisk_log_size":40173,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":22940,"ondisk_log_size":22940,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739213,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11588452,"kb_used_data":10433244,"kb_used_omap":8,"kb_used_meta":1155191,"kb_avail":82783388,"statfs":{"total":96636764160,"available":84770189312,"internally_reserved":0,"allocated":10683641856,"data_stored":10683337325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1182916253},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.72399999999999998,"15min":0.86499999999999999},"min":{"1min":0.23300000000000001,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":0.94399999999999995,"5min":5.7169999999999996,"15min":22.969000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.67500000000000004,"15min":0.71799999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.16500000000000001},"max":{"1min":4.7089999999999996,"5min":5.1689999999999996,"15min":22.835000000000001},"last":0.29399999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.84099999999999997,"15min":0.60399999999999998},"min":{"1min":0.19700000000000001,"5min":0.192,"15min":0.186},"max":{"1min":2.8599999999999999,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.0129999999999999,"15min":0.92000000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":5.8049999999999997,"5min":21.234999999999999,"15min":21.234999999999999},"last":0.54900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739211,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11622372,"kb_used_data":10433940,"kb_used_omap":15,"kb_used_meta":1188400,"kb_avail":82749468,"statfs":{"total":96636764160,"available":84735455232,"internally_reserved":0,"allocated":10684354560,"data_stored":10683936764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1216921990},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,8,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.748,"15min":0.79700000000000004},"min":{"1min":0.16900000000000001,"5min":0.13300000000000001,"15min":0.13},"max":{"1min":2.4729999999999999,"5min":14.682,"15min":14.682},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.5,"15min":0.48699999999999999},"min":{"1min":0.20399999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.1949999999999998,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.247}]},{"osd":2,"last update":"Sat Oct 21 05:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.67600000000000005,"15min":0.755},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":2.4140000000000001,"5min":5.7039999999999997,"15min":8.5389999999999997},"last":0.315},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.71099999999999997,"15min":0.77600000000000002},"min":{"1min":0.27900000000000003,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":2.4460000000000002,"5min":5.7400000000000002,"15min":9.1950000000000003},"last":0.34699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739213,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11592240,"kb_used_data":10433896,"kb_used_omap":12,"kb_used_meta":1158323,"kb_avail":82779600,"statfs":{"total":96636764160,"available":84766310400,"internally_reserved":0,"allocated":10684309504,"data_stored":10683944507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1186123177},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,5,0,4,3,20,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.623,"15min":0.58499999999999996},"min":{"1min":0.19400000000000001,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":11.773},"last":0.308},{"interface":"front","average":{"1min":0.996,"5min":0.66200000000000003,"15min":0.64500000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.19600000000000001},"max":{"1min":5.7080000000000002,"5min":5.7080000000000002,"15min":11.808999999999999},"last":0.33600000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:15:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":0.95199999999999996,"15min":0.72599999999999998},"min":{"1min":0.29299999999999998,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":5.6280000000000001,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.91400000000000003,"15min":0.81000000000000005},"min":{"1min":0.20999999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":4.226,"5min":9.516,"15min":10.074999999999999},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10683146240,"data_stored":10683146240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10683146240,"data_stored":10683146240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10683138048,"data_stored":10683138048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:15:36.011 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:36.011 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:36.012 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:36.012 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:36.013 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:15:56.014 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:15:56.700 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:15:56.701 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:15:56.797 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2175,"stamp":"2023-10-21T05:15:55.691124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21358192483,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":922110,"num_read_kb":3568123,"num_write":907581,"num_write_kb":3630815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63485,"ondisk_log_size":63485,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":34922984,"kb_used_data":31302160,"kb_used_omap":36,"kb_used_meta":3620763,"kb_avail":248192536,"statfs":{"total":289910292480,"available":254149156864,"internally_reserved":0,"allocated":32053411840,"data_stored":32052324516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":3707661772},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,6,15,8,23,18,10,6],"upper_bound":1024},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":610304,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1820,"num_read_kb":7192,"num_write":1773,"num_write_kb":7092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002133"},"pg_stats":[{"pgid":"2.7","version":"54'4589","reported_seq":7397,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:52.224249+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:15:52.224249+0000","last_peered":"2023-10-21T05:15:52.224249+0000","last_clean":"2023-10-21T05:15:52.224249+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:15:52.224249+0000","last_undegraded":"2023-10-21T05:15:52.224249+0000","last_fullsized":"2023-10-21T05:15:52.224249+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7604","reported_seq":12437,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:52.239178+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:15:52.239178+0000","last_peered":"2023-10-21T05:15:52.239178+0000","last_clean":"2023-10-21T05:15:52.239178+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:15:52.239178+0000","last_undegraded":"2023-10-21T05:15:52.239178+0000","last_fullsized":"2023-10-21T05:15:52.239178+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7604,"log_dups_size":0,"ondisk_log_size":7604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1555","reported_seq":2570,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:51.629301+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:15:51.629301+0000","last_peered":"2023-10-21T05:15:51.629301+0000","last_clean":"2023-10-21T05:15:51.629301+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:15:51.629301+0000","last_undegraded":"2023-10-21T05:15:51.629301+0000","last_fullsized":"2023-10-21T05:15:51.629301+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1572","reported_seq":2708,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:51.629313+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:15:51.629313+0000","last_peered":"2023-10-21T05:15:51.629313+0000","last_clean":"2023-10-21T05:15:51.629313+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:15:51.629313+0000","last_undegraded":"2023-10-21T05:15:51.629313+0000","last_fullsized":"2023-10-21T05:15:51.629313+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1572,"log_dups_size":0,"ondisk_log_size":1572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'235512","reported_seq":472643,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:53.621418+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:15:53.621418+0000","last_peered":"2023-10-21T05:15:53.621418+0000","last_clean":"2023-10-21T05:15:53.621418+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:15:53.621418+0000","last_undegraded":"2023-10-21T05:15:53.621418+0000","last_fullsized":"2023-10-21T05:15:53.621418+0000","mapping_epoch":17,"log_start":"54'225440","ondisk_log_start":"54'225440","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":742,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 269s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5522272256,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":236908,"num_read_kb":921180,"num_write":235512,"num_write_kb":942048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1637","reported_seq":2661,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:50.313344+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:15:50.313344+0000","last_peered":"2023-10-21T05:15:50.313344+0000","last_clean":"2023-10-21T05:15:50.313344+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:15:50.313344+0000","last_undegraded":"2023-10-21T05:15:50.313344+0000","last_fullsized":"2023-10-21T05:15:50.313344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1637,"log_dups_size":0,"ondisk_log_size":1637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'219720","reported_seq":442233,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:52.192530+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:15:52.192530+0000","last_peered":"2023-10-21T05:15:52.192530+0000","last_clean":"2023-10-21T05:15:52.192530+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:15:52.192530+0000","last_undegraded":"2023-10-21T05:15:52.192530+0000","last_fullsized":"2023-10-21T05:15:52.192530+0000","mapping_epoch":17,"log_start":"54'209627","ondisk_log_start":"54'209627","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":854,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 257s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190164480,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":222270,"num_read_kb":863792,"num_write":219720,"num_write_kb":878880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1024,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:45.202067+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:15:45.202067+0000","last_peered":"2023-10-21T05:15:45.202067+0000","last_clean":"2023-10-21T05:15:45.202067+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:15:45.202067+0000","last_undegraded":"2023-10-21T05:15:45.202067+0000","last_fullsized":"2023-10-21T05:15:45.202067+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'228629","reported_seq":459706,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:53.583741+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:15:53.583741+0000","last_peered":"2023-10-21T05:15:53.583741+0000","last_clean":"2023-10-21T05:15:53.583741+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:15:53.583741+0000","last_undegraded":"2023-10-21T05:15:53.583741+0000","last_fullsized":"2023-10-21T05:15:53.583741+0000","mapping_epoch":17,"log_start":"54'218600","ondisk_log_start":"54'218600","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":835,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 268s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5384904704,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":230844,"num_read_kb":898144,"num_write":228629,"num_write_kb":914516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3785,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:50.318081+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:15:50.318081+0000","last_peered":"2023-10-21T05:15:50.318081+0000","last_clean":"2023-10-21T05:15:50.318081+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:15:50.318081+0000","last_undegraded":"2023-10-21T05:15:50.318081+0000","last_fullsized":"2023-10-21T05:15:50.318081+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'223320","reported_seq":449751,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:52.287754+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:15:52.287754+0000","last_peered":"2023-10-21T05:15:52.287754+0000","last_clean":"2023-10-21T05:15:52.287754+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:15:52.287754+0000","last_undegraded":"2023-10-21T05:15:52.287754+0000","last_fullsized":"2023-10-21T05:15:52.287754+0000","mapping_epoch":17,"log_start":"54'213228","ondisk_log_start":"54'213228","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5260259347,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":226322,"num_read_kb":880100,"num_write":223320,"num_write_kb":893274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6053","reported_seq":9733,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:15:52.258934+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:15:52.258934+0000","last_peered":"2023-10-21T05:15:52.258934+0000","last_clean":"2023-10-21T05:15:52.258934+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:15:52.258934+0000","last_undegraded":"2023-10-21T05:15:52.258934+0000","last_fullsized":"2023-10-21T05:15:52.258934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6053,"log_dups_size":0,"ondisk_log_size":6053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":225,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:11:59.213787+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:11:59.213787+0000","last_peered":"2023-10-21T05:11:59.213787+0000","last_clean":"2023-10-21T05:11:59.213787+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:11:59.213787+0000","last_undegraded":"2023-10-21T05:11:59.213787+0000","last_fullsized":"2023-10-21T05:11:59.213787+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21357600787,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":916344,"num_read_kb":3563216,"num_write":907181,"num_write_kb":3628718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32050565120,"data_stored":32050565120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40286,"ondisk_log_size":40286,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":23100,"ondisk_log_size":23100,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739217,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11638832,"kb_used_data":10433640,"kb_used_omap":8,"kb_used_meta":1205175,"kb_avail":82733008,"statfs":{"total":96636764160,"available":84718600192,"internally_reserved":0,"allocated":10684047360,"data_stored":10683742829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1234099869},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,9,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:41 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.60199999999999998,"15min":0.84799999999999998},"min":{"1min":0.19900000000000001,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":1.3420000000000001,"5min":4.4290000000000003,"15min":22.969000000000001},"last":0.502},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.67500000000000004,"15min":0.72999999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.17899999999999999},"max":{"1min":3.778,"5min":5.1689999999999996,"15min":22.835000000000001},"last":0.27100000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.84099999999999997,"15min":0.60399999999999998},"min":{"1min":0.19700000000000001,"5min":0.192,"15min":0.186},"max":{"1min":2.8599999999999999,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.0129999999999999,"15min":0.92000000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":5.8049999999999997,"5min":21.234999999999999,"15min":21.234999999999999},"last":0.308}]}]},{"osd":1,"up_from":8,"seq":34359739215,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11659640,"kb_used_data":10434288,"kb_used_omap":15,"kb_used_meta":1225328,"kb_avail":82712200,"statfs":{"total":96636764160,"available":84697292800,"internally_reserved":0,"allocated":10684710912,"data_stored":10684293116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1254736262},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,4,5,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.748,"15min":0.79700000000000004},"min":{"1min":0.16900000000000001,"5min":0.13300000000000001,"15min":0.13},"max":{"1min":2.4729999999999999,"5min":14.682,"15min":14.682},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.5,"15min":0.48699999999999999},"min":{"1min":0.20399999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.1949999999999998,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.25}]},{"osd":2,"last update":"Sat Oct 21 05:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.67600000000000005,"15min":0.755},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":2.4140000000000001,"5min":5.7039999999999997,"15min":8.5389999999999997},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.71099999999999997,"15min":0.77600000000000002},"min":{"1min":0.27900000000000003,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":2.4460000000000002,"5min":5.7400000000000002,"15min":9.1950000000000003},"last":0.31900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739217,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11624512,"kb_used_data":10434232,"kb_used_omap":12,"kb_used_meta":1190259,"kb_avail":82747328,"statfs":{"total":96636764160,"available":84733263872,"internally_reserved":0,"allocated":10684653568,"data_stored":10684288571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1218825641},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,3,2,16,9,7,6],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.623,"15min":0.58499999999999996},"min":{"1min":0.19400000000000001,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":11.773},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.996,"5min":0.66200000000000003,"15min":0.64500000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.19600000000000001},"max":{"1min":5.7080000000000002,"5min":5.7080000000000002,"15min":11.808999999999999},"last":0.36099999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:15:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":0.95199999999999996,"15min":0.72599999999999998},"min":{"1min":0.29299999999999998,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":5.6280000000000001,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":2.653},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.91400000000000003,"15min":0.81000000000000005},"min":{"1min":0.20999999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":4.226,"5min":9.516,"15min":10.074999999999999},"last":2.6309999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10683510784,"data_stored":10683510784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10683523072,"data_stored":10683523072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10683531264,"data_stored":10683531264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:15:56.799 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:56.799 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:56.800 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:56.800 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:56.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:15:57.407 INFO:teuthology.orchestra.run.smithi172.stderr:instructing pg 3.3s0 on osd.0 to deep-scrub 2023-10-21T05:15:57.457 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:15:58.138 INFO:teuthology.orchestra.run.smithi172.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2023-10-21T05:15:58.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:15:58.735 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T05:15:58.730+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T05:15:58.989 INFO:teuthology.orchestra.run.smithi172.stderr:instructing pg 3.1s0 on osd.0 to deep-scrub 2023-10-21T05:15:59.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:15:59.877 INFO:teuthology.orchestra.run.smithi172.stderr:instructing pg 3.2s0 on osd.1 to deep-scrub 2023-10-21T05:15:59.921 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:16:19.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:16:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:16:20.699 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:16:20.782 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2187,"stamp":"2023-10-21T05:16:19.695432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21359417187,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":926749,"num_read_kb":3586575,"num_write":912105,"num_write_kb":3648911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63582,"ondisk_log_size":63582,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":34982892,"kb_used_data":31303764,"kb_used_omap":36,"kb_used_meta":3679067,"kb_avail":248132628,"statfs":{"total":289910292480,"available":254087811072,"internally_reserved":0,"allocated":32055054336,"data_stored":32053967012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":3767365068},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,6,8,8,15,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":614400,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2310,"num_read_kb":9184,"num_write":2277,"num_write_kb":9108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002140"},"pg_stats":[{"pgid":"2.7","version":"54'4616","reported_seq":7439,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:16.902655+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:16:16.902655+0000","last_peered":"2023-10-21T05:16:16.902655+0000","last_clean":"2023-10-21T05:16:16.902655+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:16:16.902655+0000","last_undegraded":"2023-10-21T05:16:16.902655+0000","last_fullsized":"2023-10-21T05:16:16.902655+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7649","reported_seq":12507,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:16.930037+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:16:16.930037+0000","last_peered":"2023-10-21T05:16:16.930037+0000","last_clean":"2023-10-21T05:16:16.930037+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:16:16.930037+0000","last_undegraded":"2023-10-21T05:16:16.930037+0000","last_fullsized":"2023-10-21T05:16:16.930037+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7649,"log_dups_size":0,"ondisk_log_size":7649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1564","reported_seq":2584,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:15.913911+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:16:15.913911+0000","last_peered":"2023-10-21T05:16:15.913911+0000","last_clean":"2023-10-21T05:16:15.913911+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:16:15.913911+0000","last_undegraded":"2023-10-21T05:16:15.913911+0000","last_fullsized":"2023-10-21T05:16:15.913911+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1564,"log_dups_size":0,"ondisk_log_size":1564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1581","reported_seq":2722,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:15.913680+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:16:15.913680+0000","last_peered":"2023-10-21T05:16:15.913680+0000","last_clean":"2023-10-21T05:16:15.913680+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:16:15.913680+0000","last_undegraded":"2023-10-21T05:16:15.913680+0000","last_fullsized":"2023-10-21T05:16:15.913680+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1581,"log_dups_size":0,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'236649","reported_seq":474984,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:18.950757+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:16:18.950757+0000","last_peered":"2023-10-21T05:16:18.950757+0000","last_clean":"2023-10-21T05:16:18.950757+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:16:18.950757+0000","last_undegraded":"2023-10-21T05:16:18.950757+0000","last_fullsized":"2023-10-21T05:16:18.950757+0000","mapping_epoch":17,"log_start":"54'226557","ondisk_log_start":"54'226557","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":820,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 294s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5522518016,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":238101,"num_read_kb":925904,"num_write":236649,"num_write_kb":946596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1650","reported_seq":2679,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:18.526763+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:16:18.526763+0000","last_peered":"2023-10-21T05:16:18.526763+0000","last_clean":"2023-10-21T05:16:18.526763+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:16:18.526763+0000","last_undegraded":"2023-10-21T05:16:18.526763+0000","last_fullsized":"2023-10-21T05:16:18.526763+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1650,"log_dups_size":0,"ondisk_log_size":1650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'220804","reported_seq":444446,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:17.537482+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:16:17.537482+0000","last_peered":"2023-10-21T05:16:17.537482+0000","last_clean":"2023-10-21T05:16:17.537482+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:16:17.537482+0000","last_undegraded":"2023-10-21T05:16:17.537482+0000","last_fullsized":"2023-10-21T05:16:17.537482+0000","mapping_epoch":17,"log_start":"54'210749","ondisk_log_start":"54'210749","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":953,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 282s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190365184,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":223384,"num_read_kb":868232,"num_write":220804,"num_write_kb":883216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1029,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:10.794697+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:16:10.794697+0000","last_peered":"2023-10-21T05:16:10.794697+0000","last_clean":"2023-10-21T05:16:10.794697+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:16:10.794697+0000","last_undegraded":"2023-10-21T05:16:10.794697+0000","last_fullsized":"2023-10-21T05:16:10.794697+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'229763","reported_seq":462029,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:18.947405+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:16:18.947405+0000","last_peered":"2023-10-21T05:16:18.947405+0000","last_clean":"2023-10-21T05:16:18.947405+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:16:18.947405+0000","last_undegraded":"2023-10-21T05:16:18.947405+0000","last_fullsized":"2023-10-21T05:16:18.947405+0000","mapping_epoch":17,"log_start":"54'219716","ondisk_log_start":"54'219716","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":898,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 293s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5385388032,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":232019,"num_read_kb":902820,"num_write":229763,"num_write_kb":919052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3801,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:10.813012+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:16:10.813012+0000","last_peered":"2023-10-21T05:16:10.813012+0000","last_clean":"2023-10-21T05:16:10.813012+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:16:10.813012+0000","last_undegraded":"2023-10-21T05:16:10.813012+0000","last_fullsized":"2023-10-21T05:16:10.813012+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'224489","reported_seq":452078,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:17.580987+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:16:17.580987+0000","last_peered":"2023-10-21T05:16:17.580987+0000","last_clean":"2023-10-21T05:16:17.580987+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:16:17.580987+0000","last_undegraded":"2023-10-21T05:16:17.580987+0000","last_fullsized":"2023-10-21T05:16:17.580987+0000","mapping_epoch":17,"log_start":"54'214439","ondisk_log_start":"54'214439","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5260554259,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":227479,"num_read_kb":884712,"num_write":224489,"num_write_kb":897950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6089","reported_seq":9791,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:16.949076+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:16:16.949076+0000","last_peered":"2023-10-21T05:16:16.949076+0000","last_clean":"2023-10-21T05:16:16.949076+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:16:16.949076+0000","last_undegraded":"2023-10-21T05:16:16.949076+0000","last_fullsized":"2023-10-21T05:16:16.949076+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":226,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:00.724406+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:14:00.724406+0000","last_peered":"2023-10-21T05:14:00.724406+0000","last_clean":"2023-10-21T05:14:00.724406+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:14:00.724406+0000","last_undegraded":"2023-10-21T05:14:00.724406+0000","last_fullsized":"2023-10-21T05:14:00.724406+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21358825491,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":920983,"num_read_kb":3581668,"num_write":911705,"num_write_kb":3646814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32052199424,"data_stored":32052199424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40244,"ondisk_log_size":40244,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":23239,"ondisk_log_size":23239,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739221,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11657328,"kb_used_data":10434088,"kb_used_omap":8,"kb_used_meta":1223223,"kb_avail":82714512,"statfs":{"total":96636764160,"available":84699660288,"internally_reserved":0,"allocated":10684506112,"data_stored":10684201581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1252581021},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:41 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.60199999999999998,"15min":0.84799999999999998},"min":{"1min":0.19900000000000001,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":1.3420000000000001,"5min":4.4290000000000003,"15min":22.969000000000001},"last":0.312},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.67500000000000004,"15min":0.72999999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.17899999999999999},"max":{"1min":3.778,"5min":5.1689999999999996,"15min":22.835000000000001},"last":0.26400000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.84099999999999997,"15min":0.60399999999999998},"min":{"1min":0.19700000000000001,"5min":0.192,"15min":0.186},"max":{"1min":2.8599999999999999,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.0129999999999999,"15min":0.92000000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":5.8049999999999997,"5min":21.234999999999999,"15min":21.234999999999999},"last":0.29599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739220,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11678240,"kb_used_data":10434840,"kb_used_omap":15,"kb_used_meta":1243376,"kb_avail":82693600,"statfs":{"total":96636764160,"available":84678246400,"internally_reserved":0,"allocated":10685276160,"data_stored":10684858364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1273217414},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.748,"15min":0.79700000000000004},"min":{"1min":0.16900000000000001,"5min":0.13300000000000001,"15min":0.13},"max":{"1min":2.4729999999999999,"5min":14.682,"15min":14.682},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.5,"15min":0.48699999999999999},"min":{"1min":0.20399999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.1949999999999998,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.25700000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.67600000000000005,"15min":0.755},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":2.4140000000000001,"5min":5.7039999999999997,"15min":8.5389999999999997},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.71099999999999997,"15min":0.77600000000000002},"min":{"1min":0.27900000000000003,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":2.4460000000000002,"5min":5.7400000000000002,"15min":9.1950000000000003},"last":0.442}]}]},{"osd":0,"up_from":8,"seq":34359739222,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11647324,"kb_used_data":10434836,"kb_used_omap":12,"kb_used_meta":1212467,"kb_avail":82724516,"statfs":{"total":96636764160,"available":84709904384,"internally_reserved":0,"allocated":10685272064,"data_stored":10684907067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1241566633},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6,2,13,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.623,"15min":0.58499999999999996},"min":{"1min":0.19400000000000001,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":11.773},"last":1.4219999999999999},{"interface":"front","average":{"1min":0.996,"5min":0.66200000000000003,"15min":0.64500000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.19600000000000001},"max":{"1min":5.7080000000000002,"5min":5.7080000000000002,"15min":11.808999999999999},"last":1.3819999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:15:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":0.95199999999999996,"15min":0.72599999999999998},"min":{"1min":0.29299999999999998,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":5.6280000000000001,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.442},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.91400000000000003,"15min":0.81000000000000005},"min":{"1min":0.20999999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":4.226,"5min":9.516,"15min":10.074999999999999},"last":0.47799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10684080128,"data_stored":10684080128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10684076032,"data_stored":10684076032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10684043264,"data_stored":10684043264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:16:20.784 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:20.784 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:20.785 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:20.785 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:20.785 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:16:35.111 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-21T05:16:35.105+0000 7f70a121e640 -1 reset not still connected to 0x55b4e884cd00 2023-10-21T05:16:40.786 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:16:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:16:41.610 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:16:41.694 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2197,"stamp":"2023-10-21T05:16:39.699021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21360310115,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":930394,"num_read_kb":3601107,"num_write":915748,"num_write_kb":3663483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63536,"ondisk_log_size":63536,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35056416,"kb_used_data":31305140,"kb_used_omap":36,"kb_used_meta":3751195,"kb_avail":248059104,"statfs":{"total":289910292480,"available":254012522496,"internally_reserved":0,"allocated":32056463360,"data_stored":32055376036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":3841224140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,13,4,6,7,6,8,16,10,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":495616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2304,"num_read_kb":9184,"num_write":2296,"num_write_kb":9184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002166"},"pg_stats":[{"pgid":"2.7","version":"54'4625","reported_seq":7460,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:36.393959+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:16:36.393959+0000","last_peered":"2023-10-21T05:16:36.393959+0000","last_clean":"2023-10-21T05:16:36.393959+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:16:36.393959+0000","last_undegraded":"2023-10-21T05:16:36.393959+0000","last_fullsized":"2023-10-21T05:16:36.393959+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7664","reported_seq":12542,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:36.394280+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:16:36.394280+0000","last_peered":"2023-10-21T05:16:36.394280+0000","last_clean":"2023-10-21T05:16:36.394280+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:16:36.394280+0000","last_undegraded":"2023-10-21T05:16:36.394280+0000","last_fullsized":"2023-10-21T05:16:36.394280+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7664,"log_dups_size":0,"ondisk_log_size":7664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1567","reported_seq":2591,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:35.914574+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:16:35.914574+0000","last_peered":"2023-10-21T05:16:35.914574+0000","last_clean":"2023-10-21T05:16:35.914574+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:16:35.914574+0000","last_undegraded":"2023-10-21T05:16:35.914574+0000","last_fullsized":"2023-10-21T05:16:35.914574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1567,"log_dups_size":0,"ondisk_log_size":1567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1584","reported_seq":2729,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:35.914580+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:16:35.914580+0000","last_peered":"2023-10-21T05:16:35.914580+0000","last_clean":"2023-10-21T05:16:35.914580+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:16:35.914580+0000","last_undegraded":"2023-10-21T05:16:35.914580+0000","last_fullsized":"2023-10-21T05:16:35.914580+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'237594","reported_seq":476911,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:39.268732+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:16:39.268732+0000","last_peered":"2023-10-21T05:16:39.268732+0000","last_clean":"2023-10-21T05:16:39.268732+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:16:39.268732+0000","last_undegraded":"2023-10-21T05:16:39.268732+0000","last_fullsized":"2023-10-21T05:16:39.268732+0000","mapping_epoch":17,"log_start":"54'227575","ondisk_log_start":"54'227575","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":892,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 315s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5522812928,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":239072,"num_read_kb":929764,"num_write":237594,"num_write_kb":950376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1656","reported_seq":2690,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:36.398951+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:16:36.398951+0000","last_peered":"2023-10-21T05:16:36.398951+0000","last_clean":"2023-10-21T05:16:36.398951+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:16:36.398951+0000","last_undegraded":"2023-10-21T05:16:36.398951+0000","last_fullsized":"2023-10-21T05:16:36.398951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1656,"log_dups_size":0,"ondisk_log_size":1656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'221697","reported_seq":446197,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:37.791591+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:16:37.791591+0000","last_peered":"2023-10-21T05:16:37.791591+0000","last_clean":"2023-10-21T05:16:37.791591+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:16:37.791591+0000","last_undegraded":"2023-10-21T05:16:37.791591+0000","last_fullsized":"2023-10-21T05:16:37.791591+0000","mapping_epoch":17,"log_start":"54'211660","ondisk_log_start":"54'211660","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1018,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 302s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190664192,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":224234,"num_read_kb":871628,"num_write":221697,"num_write_kb":886788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1033,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:26.156942+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:16:26.156942+0000","last_peered":"2023-10-21T05:16:26.156942+0000","last_clean":"2023-10-21T05:16:26.156942+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:16:26.156942+0000","last_undegraded":"2023-10-21T05:16:26.156942+0000","last_fullsized":"2023-10-21T05:16:26.156942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'230665","reported_seq":463927,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:39.269365+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:16:39.269365+0000","last_peered":"2023-10-21T05:16:39.269365+0000","last_clean":"2023-10-21T05:16:39.269365+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:16:39.269365+0000","last_undegraded":"2023-10-21T05:16:39.269365+0000","last_fullsized":"2023-10-21T05:16:39.269365+0000","mapping_epoch":17,"log_start":"54'220620","ondisk_log_start":"54'220620","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":983,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 314s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5385555968,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":233004,"num_read_kb":906756,"num_write":230665,"num_write_kb":922660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3817,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:31.282772+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:16:31.282772+0000","last_peered":"2023-10-21T05:16:31.282772+0000","last_clean":"2023-10-21T05:16:31.282772+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:16:31.282772+0000","last_undegraded":"2023-10-21T05:16:31.282772+0000","last_fullsized":"2023-10-21T05:16:31.282772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'225392","reported_seq":453819,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:37.788533+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:16:37.788533+0000","last_peered":"2023-10-21T05:16:37.788533+0000","last_clean":"2023-10-21T05:16:37.788533+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:16:37.788533+0000","last_undegraded":"2023-10-21T05:16:37.788533+0000","last_fullsized":"2023-10-21T05:16:37.788533+0000","mapping_epoch":17,"log_start":"54'215343","ondisk_log_start":"54'215343","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5260685331,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":228318,"num_read_kb":888052,"num_write":225392,"num_write_kb":901562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6101","reported_seq":9819,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:36.417678+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:16:36.417678+0000","last_peered":"2023-10-21T05:16:36.417678+0000","last_clean":"2023-10-21T05:16:36.417678+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:16:36.417678+0000","last_undegraded":"2023-10-21T05:16:36.417678+0000","last_fullsized":"2023-10-21T05:16:36.417678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6101,"log_dups_size":0,"ondisk_log_size":6101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":226,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:00.724406+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:14:00.724406+0000","last_peered":"2023-10-21T05:14:00.724406+0000","last_clean":"2023-10-21T05:14:00.724406+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:14:00.724406+0000","last_undegraded":"2023-10-21T05:14:00.724406+0000","last_fullsized":"2023-10-21T05:14:00.724406+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21359718419,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":924628,"num_read_kb":3596200,"num_write":915348,"num_write_kb":3661386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32053522432,"data_stored":32053522432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40150,"ondisk_log_size":40150,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":23287,"ondisk_log_size":23287,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739225,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11675752,"kb_used_data":10434528,"kb_used_omap":8,"kb_used_meta":1241207,"kb_avail":82696088,"statfs":{"total":96636764160,"available":84680794112,"internally_reserved":0,"allocated":10684956672,"data_stored":10684652141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1270996637},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:41 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.60199999999999998,"15min":0.84799999999999998},"min":{"1min":0.19900000000000001,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":1.3420000000000001,"5min":4.4290000000000003,"15min":22.969000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.67500000000000004,"15min":0.72999999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.17899999999999999},"max":{"1min":3.778,"5min":5.1689999999999996,"15min":22.835000000000001},"last":0.35099999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.75,"15min":0.60699999999999998},"min":{"1min":0.23599999999999999,"5min":0.192,"15min":0.186},"max":{"1min":3.5430000000000001,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.317},{"interface":"front","average":{"1min":0.623,"5min":0.877,"15min":0.92700000000000005},"min":{"1min":0.251,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":3.4609999999999999,"5min":21.234999999999999,"15min":21.234999999999999},"last":0.38}]}]},{"osd":1,"up_from":8,"seq":34359739224,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11696772,"kb_used_data":10435324,"kb_used_omap":15,"kb_used_meta":1261424,"kb_avail":82675068,"statfs":{"total":96636764160,"available":84659269632,"internally_reserved":0,"allocated":10685771776,"data_stored":10685353980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1291698566},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,12,1,2,3,4,4,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.17499999999999999,"5min":0.13300000000000001,"15min":0.13},"max":{"1min":4.6760000000000002,"5min":14.682,"15min":14.682},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.374,"5min":0.51200000000000001,"15min":0.49399999999999999},"min":{"1min":0.22500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.7529999999999999,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.40699999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.64700000000000002,"15min":0.73899999999999999},"min":{"1min":0.28000000000000003,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":0.89500000000000002,"5min":5.7039999999999997,"15min":8.5389999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.68799999999999994,"15min":0.76000000000000001},"min":{"1min":0.31900000000000001,"5min":0.27800000000000002,"15min":0.20599999999999999},"max":{"1min":0.93300000000000005,"5min":5.7400000000000002,"15min":9.1950000000000003},"last":0.46200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739226,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11683892,"kb_used_data":10435288,"kb_used_omap":12,"kb_used_meta":1248563,"kb_avail":82687948,"statfs":{"total":96636764160,"available":84672458752,"internally_reserved":0,"allocated":10685734912,"data_stored":10685369915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1278528937},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,3,3,0,4,10,7,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.64200000000000002,"15min":0.58199999999999996},"min":{"1min":0.222,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":1.4219999999999999,"5min":5.6760000000000002,"15min":11.773},"last":0.25},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.68000000000000005,"15min":0.629},"min":{"1min":0.254,"5min":0.22700000000000001,"15min":0.19600000000000001},"max":{"1min":1.3819999999999999,"5min":5.7080000000000002,"15min":11.808999999999999},"last":0.28100000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.95999999999999996,"15min":0.748},"min":{"1min":0.26100000000000001,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.653,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.83299999999999996,"15min":0.79900000000000004},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.17599999999999999},"max":{"1min":4.6289999999999996,"5min":9.516,"15min":9.516},"last":0.36799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10684518400,"data_stored":10684518400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10684563456,"data_stored":10684563456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10684440576,"data_stored":10684440576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:16:41.696 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:41.696 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:41.696 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:41.697 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:41.697 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:01.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:17:02.602 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:17:02.603 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:17:02.642 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2208,"stamp":"2023-10-21T05:17:01.703792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21360760675,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":934161,"num_read_kb":3616155,"num_write":919370,"num_write_kb":3677971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63621,"ondisk_log_size":63621,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35186868,"kb_used_data":31305860,"kb_used_omap":36,"kb_used_meta":3880923,"kb_avail":247928652,"statfs":{"total":289910292480,"available":253878939648,"internally_reserved":0,"allocated":32057200640,"data_stored":32056113316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":3974065612},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,3,20,12,10,8,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":270336,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1826,"num_read_kb":7296,"num_write":1818,"num_write_kb":7272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002299"},"pg_stats":[{"pgid":"2.7","version":"54'4643","reported_seq":7490,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:57.052359+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:16:57.052359+0000","last_peered":"2023-10-21T05:16:57.052359+0000","last_clean":"2023-10-21T05:16:57.052359+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:16:57.052359+0000","last_undegraded":"2023-10-21T05:16:57.052359+0000","last_fullsized":"2023-10-21T05:16:57.052359+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4643,"log_dups_size":0,"ondisk_log_size":4643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7694","reported_seq":12592,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:57.083655+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:16:57.083655+0000","last_peered":"2023-10-21T05:16:57.083655+0000","last_clean":"2023-10-21T05:16:57.083655+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:16:57.083655+0000","last_undegraded":"2023-10-21T05:16:57.083655+0000","last_fullsized":"2023-10-21T05:16:57.083655+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7694,"log_dups_size":0,"ondisk_log_size":7694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1573","reported_seq":2602,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:57.051430+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:16:57.051430+0000","last_peered":"2023-10-21T05:16:57.051430+0000","last_clean":"2023-10-21T05:16:57.051430+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:16:57.051430+0000","last_undegraded":"2023-10-21T05:16:57.051430+0000","last_fullsized":"2023-10-21T05:16:57.051430+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1590","reported_seq":2740,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:56.874796+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:16:56.874796+0000","last_peered":"2023-10-21T05:16:56.874796+0000","last_clean":"2023-10-21T05:16:56.874796+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:16:56.874796+0000","last_undegraded":"2023-10-21T05:16:56.874796+0000","last_fullsized":"2023-10-21T05:16:56.874796+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1590,"log_dups_size":0,"ondisk_log_size":1590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'238545","reported_seq":478825,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:59.623655+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:16:59.623655+0000","last_peered":"2023-10-21T05:16:59.623655+0000","last_clean":"2023-10-21T05:16:59.623655+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:16:59.623655+0000","last_undegraded":"2023-10-21T05:16:59.623655+0000","last_fullsized":"2023-10-21T05:16:59.623655+0000","mapping_epoch":17,"log_start":"54'228482","ondisk_log_start":"54'228482","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":938,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 335s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5522866176,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":240031,"num_read_kb":933596,"num_write":238545,"num_write_kb":954180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1660","reported_seq":2698,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:56.110413+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:16:56.110413+0000","last_peered":"2023-10-21T05:16:56.110413+0000","last_clean":"2023-10-21T05:16:56.110413+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:16:56.110413+0000","last_undegraded":"2023-10-21T05:16:56.110413+0000","last_fullsized":"2023-10-21T05:16:56.110413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1660,"log_dups_size":0,"ondisk_log_size":1660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'222592","reported_seq":448092,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:58.029549+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:16:58.029549+0000","last_peered":"2023-10-21T05:16:58.029549+0000","last_clean":"2023-10-21T05:16:58.029549+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:16:58.029549+0000","last_undegraded":"2023-10-21T05:16:58.029549+0000","last_fullsized":"2023-10-21T05:16:58.029549+0000","mapping_epoch":17,"log_start":"54'212566","ondisk_log_start":"54'212566","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1063,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 323s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190705152,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":225224,"num_read_kb":875584,"num_write":222592,"num_write_kb":890368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1038,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:51.771228+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:16:51.771228+0000","last_peered":"2023-10-21T05:16:51.771228+0000","last_clean":"2023-10-21T05:16:51.771228+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:16:51.771228+0000","last_undegraded":"2023-10-21T05:16:51.771228+0000","last_fullsized":"2023-10-21T05:16:51.771228+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'231573","reported_seq":465752,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:59.567776+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:16:59.567776+0000","last_peered":"2023-10-21T05:16:59.567776+0000","last_clean":"2023-10-21T05:16:59.567776+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:16:59.567776+0000","last_undegraded":"2023-10-21T05:16:59.567776+0000","last_fullsized":"2023-10-21T05:16:59.567776+0000","mapping_epoch":17,"log_start":"54'221528","ondisk_log_start":"54'221528","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1036,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 334s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5385719808,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":233911,"num_read_kb":910376,"num_write":231573,"num_write_kb":926292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3837,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:56.873621+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:16:56.873621+0000","last_peered":"2023-10-21T05:16:56.873621+0000","last_clean":"2023-10-21T05:16:56.873621+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:16:56.873621+0000","last_undegraded":"2023-10-21T05:16:56.873621+0000","last_fullsized":"2023-10-21T05:16:56.873621+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'226260","reported_seq":455600,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:58.029231+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:16:58.029231+0000","last_peered":"2023-10-21T05:16:58.029231+0000","last_clean":"2023-10-21T05:16:58.029231+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:16:58.029231+0000","last_undegraded":"2023-10-21T05:16:58.029231+0000","last_fullsized":"2023-10-21T05:16:58.029231+0000","mapping_epoch":17,"log_start":"54'216247","ondisk_log_start":"54'216247","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5260877843,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":229229,"num_read_kb":891692,"num_write":226260,"num_write_kb":905034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6125","reported_seq":9860,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:57.140772+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:16:57.140772+0000","last_peered":"2023-10-21T05:16:57.140772+0000","last_clean":"2023-10-21T05:16:57.140772+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:16:57.140772+0000","last_undegraded":"2023-10-21T05:16:57.140772+0000","last_fullsized":"2023-10-21T05:16:57.140772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":226,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:00.724406+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:14:00.724406+0000","last_peered":"2023-10-21T05:14:00.724406+0000","last_clean":"2023-10-21T05:14:00.724406+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:14:00.724406+0000","last_undegraded":"2023-10-21T05:14:00.724406+0000","last_fullsized":"2023-10-21T05:14:00.724406+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21360168979,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":928395,"num_read_kb":3611248,"num_write":918970,"num_write_kb":3675874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32054325248,"data_stored":32054325248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40147,"ondisk_log_size":40147,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":23375,"ondisk_log_size":23375,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739230,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11730996,"kb_used_data":10434844,"kb_used_omap":8,"kb_used_meta":1296119,"kb_avail":82640844,"statfs":{"total":96636764160,"available":84624224256,"internally_reserved":0,"allocated":10685280256,"data_stored":10684975725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1327226525},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.61299999999999999,"15min":0.84999999999999998},"min":{"1min":0.19500000000000001,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":6.625,"5min":6.625,"15min":22.969000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61199999999999999,"15min":0.73299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.4159999999999999,"5min":5.1689999999999996,"15min":22.835000000000001},"last":0.58199999999999996}]},{"osd":1,"last update":"Sat Oct 21 05:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.75,"15min":0.60699999999999998},"min":{"1min":0.23599999999999999,"5min":0.192,"15min":0.186},"max":{"1min":3.5430000000000001,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.877,"15min":0.92700000000000005},"min":{"1min":0.251,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":3.4609999999999999,"5min":21.234999999999999,"15min":21.234999999999999},"last":0.32200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739228,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11733092,"kb_used_data":10435548,"kb_used_omap":15,"kb_used_meta":1297520,"kb_avail":82638748,"statfs":{"total":96636764160,"available":84622077952,"internally_reserved":0,"allocated":10686001152,"data_stored":10685583356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1328660870},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,16,8,6,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.17499999999999999,"5min":0.13300000000000001,"15min":0.13},"max":{"1min":4.6760000000000002,"5min":14.682,"15min":14.682},"last":1.087},{"interface":"front","average":{"1min":0.374,"5min":0.51200000000000001,"15min":0.49399999999999999},"min":{"1min":0.22500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.7529999999999999,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.29399999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.64700000000000002,"15min":0.73899999999999999},"min":{"1min":0.28000000000000003,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":0.89500000000000002,"5min":5.7039999999999997,"15min":8.5389999999999997},"last":0.314},{"interface":"front","average":{"1min":0.438,"5min":0.68799999999999994,"15min":0.76000000000000001},"min":{"1min":0.31900000000000001,"5min":0.27800000000000002,"15min":0.20599999999999999},"max":{"1min":0.93300000000000005,"5min":5.7400000000000002,"15min":9.1950000000000003},"last":0.34399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739230,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11722780,"kb_used_data":10435468,"kb_used_omap":12,"kb_used_meta":1287283,"kb_avail":82649060,"statfs":{"total":96636764160,"available":84632637440,"internally_reserved":0,"allocated":10685919232,"data_stored":10685554235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1318178217},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,4,4,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.64200000000000002,"15min":0.58199999999999996},"min":{"1min":0.222,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":1.4219999999999999,"5min":5.6760000000000002,"15min":11.773},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.68000000000000005,"15min":0.629},"min":{"1min":0.254,"5min":0.22700000000000001,"15min":0.19600000000000001},"max":{"1min":1.3819999999999999,"5min":5.7080000000000002,"15min":11.808999999999999},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.95999999999999996,"15min":0.748},"min":{"1min":0.26100000000000001,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.653,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.83299999999999996,"15min":0.79900000000000004},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.17599999999999999},"max":{"1min":4.6289999999999996,"5min":9.516,"15min":9.516},"last":0.33800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10684772352,"data_stored":10684772352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10684760064,"data_stored":10684760064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10684792832,"data_stored":10684792832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:17:02.644 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:02.645 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:02.645 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:02.645 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:02.646 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:10.807 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.804+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T05:17:10.807 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.804+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:17:10.813 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.808+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T05:17:10.814 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.808+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:17:10.814 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.808+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T05:17:10.814 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.808+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:17:10.815 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.808+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:17:10.815 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.808+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:17:10.816 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.812+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:17:10.817 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.812+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T05:17:10.817 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.812+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T05:17:10.817 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.812+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:17:10.817 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:10.812+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:17:22.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:17:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:17:23.557 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:17:23.599 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2218,"stamp":"2023-10-21T05:17:21.713001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21361055587,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":938054,"num_read_kb":3631663,"num_write":922909,"num_write_kb":3692127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63891,"ondisk_log_size":63891,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35273088,"kb_used_data":31306260,"kb_used_omap":36,"kb_used_meta":3966747,"kb_avail":247842432,"statfs":{"total":289910292480,"available":253790650368,"internally_reserved":0,"allocated":32057610240,"data_stored":32056522916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":4061949388},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,9,14,11,15,10,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2459,"num_read_kb":9796,"num_write":2250,"num_write_kb":9000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007698"},"pg_stats":[{"pgid":"2.7","version":"54'4679","reported_seq":7538,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:17.354473+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:17:17.354473+0000","last_peered":"2023-10-21T05:17:17.354473+0000","last_clean":"2023-10-21T05:17:17.354473+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:17:17.354473+0000","last_undegraded":"2023-10-21T05:17:17.354473+0000","last_fullsized":"2023-10-21T05:17:17.354473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4679,"log_dups_size":0,"ondisk_log_size":4679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7754","reported_seq":12672,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:17.355130+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:17:17.355130+0000","last_peered":"2023-10-21T05:17:17.355130+0000","last_clean":"2023-10-21T05:17:17.355130+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:17:17.355130+0000","last_undegraded":"2023-10-21T05:17:17.355130+0000","last_fullsized":"2023-10-21T05:17:17.355130+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7754,"log_dups_size":0,"ondisk_log_size":7754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1585","reported_seq":2618,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:12.238961+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:17:12.238961+0000","last_peered":"2023-10-21T05:17:12.238961+0000","last_clean":"2023-10-21T05:17:12.238961+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:17:12.238961+0000","last_undegraded":"2023-10-21T05:17:12.238961+0000","last_fullsized":"2023-10-21T05:17:12.238961+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1585,"log_dups_size":0,"ondisk_log_size":1585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1602","reported_seq":2756,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:12.239145+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:17:12.239145+0000","last_peered":"2023-10-21T05:17:12.239145+0000","last_clean":"2023-10-21T05:17:12.239145+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:17:12.239145+0000","last_undegraded":"2023-10-21T05:17:12.239145+0000","last_fullsized":"2023-10-21T05:17:12.239145+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1602,"log_dups_size":0,"ondisk_log_size":1602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'239476","reported_seq":480751,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:19.770109+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:17:19.770109+0000","last_peered":"2023-10-21T05:17:19.770109+0000","last_clean":"2023-10-21T05:17:19.770109+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:17:19.770109+0000","last_undegraded":"2023-10-21T05:17:19.770109+0000","last_fullsized":"2023-10-21T05:17:19.770109+0000","mapping_epoch":17,"log_start":"54'229392","ondisk_log_start":"54'229392","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":967,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 355s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523013632,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":241014,"num_read_kb":937516,"num_write":239476,"num_write_kb":957904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1669","reported_seq":2711,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:17.353466+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:17:17.353466+0000","last_peered":"2023-10-21T05:17:17.353466+0000","last_clean":"2023-10-21T05:17:17.353466+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:17:17.353466+0000","last_undegraded":"2023-10-21T05:17:17.353466+0000","last_fullsized":"2023-10-21T05:17:17.353466+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1669,"log_dups_size":0,"ondisk_log_size":1669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'223450","reported_seq":449941,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:18.315258+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:17:18.315258+0000","last_peered":"2023-10-21T05:17:18.315258+0000","last_clean":"2023-10-21T05:17:18.315258+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:17:18.315258+0000","last_undegraded":"2023-10-21T05:17:18.315258+0000","last_fullsized":"2023-10-21T05:17:18.315258+0000","mapping_epoch":17,"log_start":"54'213384","ondisk_log_start":"54'213384","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1130,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 343s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190729728,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":226204,"num_read_kb":879480,"num_write":223450,"num_write_kb":893800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1042,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:12.294656+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:17:12.294656+0000","last_peered":"2023-10-21T05:17:12.294656+0000","last_clean":"2023-10-21T05:17:12.294656+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:17:12.294656+0000","last_undegraded":"2023-10-21T05:17:12.294656+0000","last_fullsized":"2023-10-21T05:17:12.294656+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'232468","reported_seq":467619,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:19.790555+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:17:19.790555+0000","last_peered":"2023-10-21T05:17:19.790555+0000","last_clean":"2023-10-21T05:17:19.790555+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:17:19.790555+0000","last_undegraded":"2023-10-21T05:17:19.790555+0000","last_fullsized":"2023-10-21T05:17:19.790555+0000","mapping_epoch":17,"log_start":"54'222439","ondisk_log_start":"54'222439","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1115,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 354s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5385744384,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":234882,"num_read_kb":914244,"num_write":232468,"num_write_kb":929872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3853,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:17.355672+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:17:17.355672+0000","last_peered":"2023-10-21T05:17:17.355672+0000","last_clean":"2023-10-21T05:17:17.355672+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:17:17.355672+0000","last_undegraded":"2023-10-21T05:17:17.355672+0000","last_fullsized":"2023-10-21T05:17:17.355672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'227115","reported_seq":457414,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:18.358879+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:17:18.358879+0000","last_peered":"2023-10-21T05:17:18.358879+0000","last_clean":"2023-10-21T05:17:18.358879+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:17:18.358879+0000","last_undegraded":"2023-10-21T05:17:18.358879+0000","last_fullsized":"2023-10-21T05:17:18.358879+0000","mapping_epoch":17,"log_start":"54'217050","ondisk_log_start":"54'217050","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5260976147,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":230188,"num_read_kb":895516,"num_write":227115,"num_write_kb":908454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6169","reported_seq":9920,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:17.360392+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:17:17.360392+0000","last_peered":"2023-10-21T05:17:17.360392+0000","last_clean":"2023-10-21T05:17:17.360392+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:17:17.360392+0000","last_undegraded":"2023-10-21T05:17:17.360392+0000","last_fullsized":"2023-10-21T05:17:17.360392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6169,"log_dups_size":0,"ondisk_log_size":6169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":226,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:00.724406+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:14:00.724406+0000","last_peered":"2023-10-21T05:14:00.724406+0000","last_clean":"2023-10-21T05:14:00.724406+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:14:00.724406+0000","last_undegraded":"2023-10-21T05:14:00.724406+0000","last_fullsized":"2023-10-21T05:14:00.724406+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21360463891,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":932288,"num_read_kb":3626756,"num_write":922509,"num_write_kb":3690030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32054771712,"data_stored":32054771712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40244,"ondisk_log_size":40244,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":23548,"ondisk_log_size":23548,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739234,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11767216,"kb_used_data":10434984,"kb_used_omap":8,"kb_used_meta":1332215,"kb_avail":82604624,"statfs":{"total":96636764160,"available":84587134976,"internally_reserved":0,"allocated":10685423616,"data_stored":10685119085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1364188829},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,7,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.61299999999999999,"15min":0.84999999999999998},"min":{"1min":0.19500000000000001,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":6.625,"5min":6.625,"15min":22.969000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61199999999999999,"15min":0.73299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.4159999999999999,"5min":5.1689999999999996,"15min":22.835000000000001},"last":0.436}]},{"osd":1,"last update":"Sat Oct 21 05:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.75,"15min":0.60699999999999998},"min":{"1min":0.23599999999999999,"5min":0.192,"15min":0.186},"max":{"1min":3.5430000000000001,"5min":19.574000000000002,"15min":19.574000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.877,"15min":0.92700000000000005},"min":{"1min":0.251,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":3.4609999999999999,"5min":21.234999999999999,"15min":21.234999999999999},"last":0.47199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739232,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11770056,"kb_used_data":10435648,"kb_used_omap":15,"kb_used_meta":1334384,"kb_avail":82601784,"statfs":{"total":96636764160,"available":84584226816,"internally_reserved":0,"allocated":10686103552,"data_stored":10685685756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1366409606},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.17499999999999999,"5min":0.13300000000000001,"15min":0.13},"max":{"1min":4.6760000000000002,"5min":14.682,"15min":14.682},"last":2.7709999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.51200000000000001,"15min":0.49399999999999999},"min":{"1min":0.22500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.7529999999999999,"5min":9.1210000000000004,"15min":9.1210000000000004},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.64700000000000002,"15min":0.73899999999999999},"min":{"1min":0.28000000000000003,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":0.89500000000000002,"5min":5.7039999999999997,"15min":8.5389999999999997},"last":0.309},{"interface":"front","average":{"1min":0.438,"5min":0.68799999999999994,"15min":0.76000000000000001},"min":{"1min":0.31900000000000001,"5min":0.27800000000000002,"15min":0.20599999999999999},"max":{"1min":0.93300000000000005,"5min":5.7400000000000002,"15min":9.1950000000000003},"last":0.33300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739234,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11735816,"kb_used_data":10435628,"kb_used_omap":12,"kb_used_meta":1300147,"kb_avail":82636024,"statfs":{"total":96636764160,"available":84619288576,"internally_reserved":0,"allocated":10686083072,"data_stored":10685718075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1331350953},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2,4,5,9,7,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.64200000000000002,"15min":0.58199999999999996},"min":{"1min":0.222,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":1.4219999999999999,"5min":5.6760000000000002,"15min":11.773},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.68000000000000005,"15min":0.629},"min":{"1min":0.254,"5min":0.22700000000000001,"15min":0.19600000000000001},"max":{"1min":1.3819999999999999,"5min":5.7080000000000002,"15min":11.808999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.95999999999999996,"15min":0.748},"min":{"1min":0.26100000000000001,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.653,"5min":9.3140000000000001,"15min":9.3140000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.83299999999999996,"15min":0.79900000000000004},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.17599999999999999},"max":{"1min":4.6289999999999996,"5min":9.516,"15min":9.516},"last":0.41899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10684948480,"data_stored":10684948480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10684911616,"data_stored":10684911616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10684911616,"data_stored":10684911616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:17:23.601 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:23.602 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:23.602 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:23.602 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:23.603 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:43.604 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:17:44.529 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:17:44.530 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:17:44.572 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2229,"stamp":"2023-10-21T05:17:43.716872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21361125219,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":941813,"num_read_kb":3646635,"num_write":926518,"num_write_kb":3706563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64040,"ondisk_log_size":64040,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35319036,"kb_used_data":31306384,"kb_used_omap":36,"kb_used_meta":4012571,"kb_avail":247796484,"statfs":{"total":289910292480,"available":253743599616,"internally_reserved":0,"allocated":32057737216,"data_stored":32056649892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":4108873164},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,5,12,9,15,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32768,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1925,"num_read_kb":7656,"num_write":1838,"num_write_kb":7352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002138"},"pg_stats":[{"pgid":"2.7","version":"54'4715","reported_seq":7587,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:38.514315+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:17:38.514315+0000","last_peered":"2023-10-21T05:17:38.514315+0000","last_clean":"2023-10-21T05:17:38.514315+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:17:38.514315+0000","last_undegraded":"2023-10-21T05:17:38.514315+0000","last_fullsized":"2023-10-21T05:17:38.514315+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4715,"log_dups_size":0,"ondisk_log_size":4715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7814","reported_seq":12753,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:38.533734+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:17:38.533734+0000","last_peered":"2023-10-21T05:17:38.533734+0000","last_clean":"2023-10-21T05:17:38.533734+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:17:38.533734+0000","last_undegraded":"2023-10-21T05:17:38.533734+0000","last_fullsized":"2023-10-21T05:17:38.533734+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7814,"log_dups_size":0,"ondisk_log_size":7814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1597","reported_seq":2634,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:37.833855+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:17:37.833855+0000","last_peered":"2023-10-21T05:17:37.833855+0000","last_clean":"2023-10-21T05:17:37.833855+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:17:37.833855+0000","last_undegraded":"2023-10-21T05:17:37.833855+0000","last_fullsized":"2023-10-21T05:17:37.833855+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1597,"log_dups_size":0,"ondisk_log_size":1597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1614","reported_seq":2772,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:37.834042+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:17:37.834042+0000","last_peered":"2023-10-21T05:17:37.834042+0000","last_clean":"2023-10-21T05:17:37.834042+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:17:37.834042+0000","last_undegraded":"2023-10-21T05:17:37.834042+0000","last_fullsized":"2023-10-21T05:17:37.834042+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1614,"log_dups_size":0,"ondisk_log_size":1614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'240385","reported_seq":482656,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:40.105003+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:17:40.105003+0000","last_peered":"2023-10-21T05:17:40.105003+0000","last_clean":"2023-10-21T05:17:40.105003+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:17:40.105003+0000","last_undegraded":"2023-10-21T05:17:40.105003+0000","last_fullsized":"2023-10-21T05:17:40.105003+0000","mapping_epoch":17,"log_start":"54'230302","ondisk_log_start":"54'230302","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1056,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 376s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523013632,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":241999,"num_read_kb":941440,"num_write":240385,"num_write_kb":961540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1673","reported_seq":2719,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:33.752791+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:17:33.752791+0000","last_peered":"2023-10-21T05:17:33.752791+0000","last_clean":"2023-10-21T05:17:33.752791+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:17:33.752791+0000","last_undegraded":"2023-10-21T05:17:33.752791+0000","last_fullsized":"2023-10-21T05:17:33.752791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1673,"log_dups_size":0,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'224367","reported_seq":451748,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:38.704125+0000","last_change":"2023-10-21T05:11:34.741689+0000","last_active":"2023-10-21T05:17:38.704125+0000","last_peered":"2023-10-21T05:17:38.704125+0000","last_clean":"2023-10-21T05:17:38.704125+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:17:38.704125+0000","last_undegraded":"2023-10-21T05:17:38.704125+0000","last_fullsized":"2023-10-21T05:17:38.704125+0000","mapping_epoch":17,"log_start":"54'214292","ondisk_log_start":"54'214292","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1225,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 363s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190737920,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":227088,"num_read_kb":882996,"num_write":224367,"num_write_kb":897468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1046,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:32.822065+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:17:32.822065+0000","last_peered":"2023-10-21T05:17:32.822065+0000","last_clean":"2023-10-21T05:17:32.822065+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:17:32.822065+0000","last_undegraded":"2023-10-21T05:17:32.822065+0000","last_fullsized":"2023-10-21T05:17:32.822065+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'233399","reported_seq":469508,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:40.091220+0000","last_change":"2023-10-21T05:11:25.084208+0000","last_active":"2023-10-21T05:17:40.091220+0000","last_peered":"2023-10-21T05:17:40.091220+0000","last_clean":"2023-10-21T05:17:40.091220+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:17:40.091220+0000","last_undegraded":"2023-10-21T05:17:40.091220+0000","last_fullsized":"2023-10-21T05:17:40.091220+0000","mapping_epoch":17,"log_start":"54'223349","ondisk_log_start":"54'223349","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1217,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 375s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5385789440,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":235823,"num_read_kb":917988,"num_write":233399,"num_write_kb":933596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3869,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:37.838935+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:17:37.838935+0000","last_peered":"2023-10-21T05:17:37.838935+0000","last_clean":"2023-10-21T05:17:37.838935+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:17:37.838935+0000","last_undegraded":"2023-10-21T05:17:37.838935+0000","last_fullsized":"2023-10-21T05:17:37.838935+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'227967","reported_seq":459213,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:38.703993+0000","last_change":"2023-10-21T04:05:28.002246+0000","last_active":"2023-10-21T05:17:38.703993+0000","last_peered":"2023-10-21T05:17:38.703993+0000","last_clean":"2023-10-21T05:17:38.703993+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:17:38.703993+0000","last_undegraded":"2023-10-21T05:17:38.703993+0000","last_fullsized":"2023-10-21T05:17:38.703993+0000","mapping_epoch":17,"log_start":"54'217954","ondisk_log_start":"54'217954","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5260992531,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":231137,"num_read_kb":899304,"num_write":227967,"num_write_kb":911862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6217","reported_seq":9984,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:38.572954+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:17:38.572954+0000","last_peered":"2023-10-21T05:17:38.572954+0000","last_clean":"2023-10-21T05:17:38.572954+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:17:38.572954+0000","last_undegraded":"2023-10-21T05:17:38.572954+0000","last_fullsized":"2023-10-21T05:17:38.572954+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6217,"log_dups_size":0,"ondisk_log_size":6217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":226,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:00.724406+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:14:00.724406+0000","last_peered":"2023-10-21T05:14:00.724406+0000","last_clean":"2023-10-21T05:14:00.724406+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:14:00.724406+0000","last_undegraded":"2023-10-21T05:14:00.724406+0000","last_fullsized":"2023-10-21T05:14:00.724406+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21360533523,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":936047,"num_read_kb":3641728,"num_write":926118,"num_write_kb":3704466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32054902784,"data_stored":32054902784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40221,"ondisk_log_size":40221,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":23720,"ondisk_log_size":23720,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739238,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11781144,"kb_used_data":10435024,"kb_used_omap":8,"kb_used_meta":1346103,"kb_avail":82590696,"statfs":{"total":96636764160,"available":84572872704,"internally_reserved":0,"allocated":10685464576,"data_stored":10685160045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1378410141},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.61299999999999999,"15min":0.84999999999999998},"min":{"1min":0.19500000000000001,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":6.625,"5min":6.625,"15min":22.969000000000001},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61199999999999999,"15min":0.73299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.4159999999999999,"5min":5.1689999999999996,"15min":22.835000000000001},"last":0.28799999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:17:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.623},"min":{"1min":0.217,"5min":0.192,"15min":0.186},"max":{"1min":4.9939999999999998,"5min":4.9939999999999998,"15min":19.574000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63,"15min":0.91900000000000004},"min":{"1min":0.252,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":1.252,"5min":5.8049999999999997,"15min":21.234999999999999},"last":0.33700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739236,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11784012,"kb_used_data":10435716,"kb_used_omap":15,"kb_used_meta":1348272,"kb_avail":82587828,"statfs":{"total":96636764160,"available":84569935872,"internally_reserved":0,"allocated":10686173184,"data_stored":10685755388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1380630918},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,4,2,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:27 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.86099999999999999,"15min":0.79700000000000004},"min":{"1min":0.17299999999999999,"5min":0.16300000000000001,"15min":0.13},"max":{"1min":3.524,"5min":14.682,"15min":14.682},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.43099999999999999,"15min":0.47099999999999997},"min":{"1min":0.23699999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.48299999999999998,"5min":2.9140000000000001,"15min":9.1210000000000004},"last":0.44500000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:17:27 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.56999999999999995,"15min":0.75900000000000001},"min":{"1min":0.25800000000000001,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":7.0789999999999997,"5min":7.0789999999999997,"15min":8.5389999999999997},"last":0.5},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.61099999999999999,"15min":0.78100000000000003},"min":{"1min":0.33300000000000002,"5min":0.27800000000000002,"15min":0.20599999999999999},"max":{"1min":7.1130000000000004,"5min":7.1130000000000004,"15min":9.1950000000000003},"last":0.57099999999999995}]}]},{"osd":0,"up_from":8,"seq":34359739238,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11753880,"kb_used_data":10435644,"kb_used_omap":12,"kb_used_meta":1318195,"kb_avail":82617960,"statfs":{"total":96636764160,"available":84600791040,"internally_reserved":0,"allocated":10686099456,"data_stored":10685734459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1349832105},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,5,2,5,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.59799999999999998,"15min":0.58399999999999996},"min":{"1min":0.188,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":2.5369999999999999,"5min":5.6760000000000002,"15min":11.773},"last":0.438},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":5.0609999999999999,"5min":5.7080000000000002,"15min":11.808999999999999},"last":0.47799999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.96099999999999997,"15min":0.77200000000000002},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":5.0800000000000001,"5min":5.6529999999999996,"15min":9.3140000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.999,"5min":0.85699999999999998,"15min":0.82199999999999995},"min":{"1min":0.28499999999999998,"5min":0.20899999999999999,"15min":0.17599999999999999},"max":{"1min":5.0469999999999997,"5min":5.0469999999999997,"15min":9.516},"last":0.33600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10684960768,"data_stored":10684960768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10684973056,"data_stored":10684973056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10684968960,"data_stored":10684968960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:17:44.574 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:44.574 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:44.574 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:44.575 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:44.575 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:51.795 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:51.791+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:17:51.811 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:51.807+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:17:51.811 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:17:51.807+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:18:04.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:18:05.211 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:18:05.211 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:18:05.253 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2239,"stamp":"2023-10-21T05:18:03.729394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21361239907,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":945712,"num_read_kb":3662163,"num_write":930125,"num_write_kb":3720991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64115,"ondisk_log_size":64115,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35408060,"kb_used_data":31306596,"kb_used_omap":36,"kb_used_meta":4101403,"kb_avail":247707460,"statfs":{"total":289910292480,"available":253652439040,"internally_reserved":0,"allocated":32057954304,"data_stored":32056866980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":4199837132},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,4,7,17,17,14,13,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":57344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1944,"num_read_kb":7748,"num_write":1822,"num_write_kb":7288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008444"},"pg_stats":[{"pgid":"2.7","version":"54'4740","reported_seq":7624,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:58.318208+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:17:58.318208+0000","last_peered":"2023-10-21T05:17:58.318208+0000","last_clean":"2023-10-21T05:17:58.318208+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:17:58.318208+0000","last_undegraded":"2023-10-21T05:17:58.318208+0000","last_fullsized":"2023-10-21T05:17:58.318208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4740,"log_dups_size":0,"ondisk_log_size":4740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7854","reported_seq":12813,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:58.319044+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:17:58.319044+0000","last_peered":"2023-10-21T05:17:58.319044+0000","last_clean":"2023-10-21T05:17:58.319044+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:17:58.319044+0000","last_undegraded":"2023-10-21T05:17:58.319044+0000","last_fullsized":"2023-10-21T05:17:58.319044+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7854,"log_dups_size":0,"ondisk_log_size":7854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1606","reported_seq":2647,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:53.197243+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:17:53.197243+0000","last_peered":"2023-10-21T05:17:53.197243+0000","last_clean":"2023-10-21T05:17:53.197243+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:17:53.197243+0000","last_undegraded":"2023-10-21T05:17:53.197243+0000","last_fullsized":"2023-10-21T05:17:53.197243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1606,"log_dups_size":0,"ondisk_log_size":1606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1623","reported_seq":2785,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:53.197734+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:17:53.197734+0000","last_peered":"2023-10-21T05:17:53.197734+0000","last_clean":"2023-10-21T05:17:53.197734+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:17:53.197734+0000","last_undegraded":"2023-10-21T05:17:53.197734+0000","last_fullsized":"2023-10-21T05:17:53.197734+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1623,"log_dups_size":0,"ondisk_log_size":1623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'241326","reported_seq":484628,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:00.364217+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:18:00.364217+0000","last_peered":"2023-10-21T05:18:00.364217+0000","last_clean":"2023-10-21T05:18:00.364217+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:18:00.364217+0000","last_undegraded":"2023-10-21T05:18:00.364217+0000","last_fullsized":"2023-10-21T05:18:00.364217+0000","mapping_epoch":17,"log_start":"54'231310","ondisk_log_start":"54'231310","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1098,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 396s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523062784,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":243024,"num_read_kb":945520,"num_write":241326,"num_write_kb":965304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1680","reported_seq":2730,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:58.356381+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:17:58.356381+0000","last_peered":"2023-10-21T05:17:58.356381+0000","last_clean":"2023-10-21T05:17:58.356381+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:17:58.356381+0000","last_undegraded":"2023-10-21T05:17:58.356381+0000","last_fullsized":"2023-10-21T05:17:58.356381+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1680,"log_dups_size":0,"ondisk_log_size":1680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'225283","reported_seq":453636,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:58.938149+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:17:58.938149+0000","last_peered":"2023-10-21T05:17:58.938149+0000","last_clean":"2023-10-21T05:17:58.938149+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:17:58.938149+0000","last_undegraded":"2023-10-21T05:17:58.938149+0000","last_fullsized":"2023-10-21T05:17:58.938149+0000","mapping_epoch":17,"log_start":"54'215203","ondisk_log_start":"54'215203","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190737920,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":228049,"num_read_kb":886816,"num_write":225283,"num_write_kb":901132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1050,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:53.193576+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:17:53.193576+0000","last_peered":"2023-10-21T05:17:53.193576+0000","last_clean":"2023-10-21T05:17:53.193576+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:17:53.193576+0000","last_undegraded":"2023-10-21T05:17:53.193576+0000","last_fullsized":"2023-10-21T05:17:53.193576+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'234278","reported_seq":471370,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:00.372147+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:18:00.372147+0000","last_peered":"2023-10-21T05:18:00.372147+0000","last_clean":"2023-10-21T05:18:00.372147+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:18:00.372147+0000","last_undegraded":"2023-10-21T05:18:00.372147+0000","last_fullsized":"2023-10-21T05:18:00.372147+0000","mapping_epoch":17,"log_start":"54'224263","ondisk_log_start":"54'224263","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5385838592,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":236789,"num_read_kb":921844,"num_write":234278,"num_write_kb":937112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3886,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:58.320895+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:17:58.320895+0000","last_peered":"2023-10-21T05:17:58.320895+0000","last_clean":"2023-10-21T05:17:58.320895+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:17:58.320895+0000","last_undegraded":"2023-10-21T05:17:58.320895+0000","last_fullsized":"2023-10-21T05:17:58.320895+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'228838","reported_seq":461039,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:58.935437+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:17:58.935437+0000","last_peered":"2023-10-21T05:17:58.935437+0000","last_clean":"2023-10-21T05:17:58.935437+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:17:58.935437+0000","last_undegraded":"2023-10-21T05:17:58.935437+0000","last_fullsized":"2023-10-21T05:17:58.935437+0000","mapping_epoch":17,"log_start":"54'218772","ondisk_log_start":"54'218772","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":53,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 14s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261008915,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":232084,"num_read_kb":903076,"num_write":228838,"num_write_kb":915346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6246","reported_seq":10029,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:17:58.331707+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:17:58.331707+0000","last_peered":"2023-10-21T05:17:58.331707+0000","last_clean":"2023-10-21T05:17:58.331707+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:17:58.331707+0000","last_undegraded":"2023-10-21T05:17:58.331707+0000","last_fullsized":"2023-10-21T05:17:58.331707+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6246,"log_dups_size":0,"ondisk_log_size":6246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":226,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:14:00.724406+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:14:00.724406+0000","last_peered":"2023-10-21T05:14:00.724406+0000","last_clean":"2023-10-21T05:14:00.724406+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:14:00.724406+0000","last_undegraded":"2023-10-21T05:14:00.724406+0000","last_fullsized":"2023-10-21T05:14:00.724406+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21360648211,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":939946,"num_read_kb":3657256,"num_write":929725,"num_write_kb":3718894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32055148544,"data_stored":32055148544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40177,"ondisk_log_size":40177,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":23839,"ondisk_log_size":23839,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739242,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11817268,"kb_used_data":10435116,"kb_used_omap":8,"kb_used_meta":1382135,"kb_avail":82554572,"statfs":{"total":96636764160,"available":84535881728,"internally_reserved":0,"allocated":10685558784,"data_stored":10685254253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1415306909},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,6,9,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:43 2023","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":0.81799999999999995,"15min":0.93600000000000005},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16800000000000001},"max":{"1min":11.904999999999999,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":0.67500000000000004,"15min":0.79400000000000004},"min":{"1min":0.23499999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.252000000000001,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.24199999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:17:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.623},"min":{"1min":0.217,"5min":0.192,"15min":0.186},"max":{"1min":4.9939999999999998,"5min":4.9939999999999998,"15min":19.574000000000002},"last":0.222},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63,"15min":0.91900000000000004},"min":{"1min":0.252,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":1.252,"5min":5.8049999999999997,"15min":21.234999999999999},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739240,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11802132,"kb_used_data":10435788,"kb_used_omap":15,"kb_used_meta":1366320,"kb_avail":82569708,"statfs":{"total":96636764160,"available":84551380992,"internally_reserved":0,"allocated":10686246912,"data_stored":10685829116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1399112070},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,0,1,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:27 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.86099999999999999,"15min":0.79700000000000004},"min":{"1min":0.17299999999999999,"5min":0.16300000000000001,"15min":0.13},"max":{"1min":3.524,"5min":14.682,"15min":14.682},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.43099999999999999,"15min":0.47099999999999997},"min":{"1min":0.23699999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.48299999999999998,"5min":2.9140000000000001,"15min":9.1210000000000004},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Oct 21 05:17:27 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.56999999999999995,"15min":0.75900000000000001},"min":{"1min":0.25800000000000001,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":7.0789999999999997,"5min":7.0789999999999997,"15min":8.5389999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.61099999999999999,"15min":0.78100000000000003},"min":{"1min":0.33300000000000002,"5min":0.27800000000000002,"15min":0.20599999999999999},"max":{"1min":7.1130000000000004,"5min":7.1130000000000004,"15min":9.1950000000000003},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739242,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11788660,"kb_used_data":10435692,"kb_used_omap":12,"kb_used_meta":1352947,"kb_avail":82583180,"statfs":{"total":96636764160,"available":84565176320,"internally_reserved":0,"allocated":10686148608,"data_stored":10685783611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1385418153},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,8,5,9,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.59799999999999998,"15min":0.58399999999999996},"min":{"1min":0.188,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":2.5369999999999999,"5min":5.6760000000000002,"15min":11.773},"last":0.253},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":5.0609999999999999,"5min":5.7080000000000002,"15min":11.808999999999999},"last":0.41099999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.96099999999999997,"15min":0.77200000000000002},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":5.0800000000000001,"5min":5.6529999999999996,"15min":9.3140000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.999,"5min":0.85699999999999998,"15min":0.82199999999999995},"min":{"1min":0.28499999999999998,"5min":0.20899999999999999,"15min":0.17599999999999999},"max":{"1min":5.0469999999999997,"5min":5.0469999999999997,"15min":9.516},"last":0.22800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685038592,"data_stored":10685038592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685054976,"data_stored":10685054976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685054976,"data_stored":10685054976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:18:05.255 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:05.256 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:05.256 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:18:24.306 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:18:24.302+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:18:24.317 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:18:24.310+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:18:24.317 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:18:24.310+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:18:25.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:18:26.032 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:18:26.033 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:18:26.077 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2250,"stamp":"2023-10-21T05:18:25.733162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21361362787,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":950641,"num_read_kb":3681763,"num_write":934893,"num_write_kb":3740063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64319,"ondisk_log_size":64319,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35525636,"kb_used_data":31306772,"kb_used_omap":36,"kb_used_meta":4218779,"kb_avail":247589884,"statfs":{"total":289910292480,"available":253532041216,"internally_reserved":0,"allocated":32058134528,"data_stored":32057047204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":4320030156},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,8,2,25,18,19,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":65536,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2955,"num_read_kb":11752,"num_write":2858,"num_write_kb":11432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002022"},"pg_stats":[{"pgid":"2.7","version":"54'4779","reported_seq":7678,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:23.914727+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:18:23.914727+0000","last_peered":"2023-10-21T05:18:23.914727+0000","last_clean":"2023-10-21T05:18:23.914727+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:18:23.914727+0000","last_undegraded":"2023-10-21T05:18:23.914727+0000","last_fullsized":"2023-10-21T05:18:23.914727+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7919","reported_seq":12903,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:23.915358+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:18:23.915358+0000","last_peered":"2023-10-21T05:18:23.915358+0000","last_clean":"2023-10-21T05:18:23.915358+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:18:23.915358+0000","last_undegraded":"2023-10-21T05:18:23.915358+0000","last_fullsized":"2023-10-21T05:18:23.915358+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7919,"log_dups_size":0,"ondisk_log_size":7919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1619","reported_seq":2665,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:22.995510+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:18:22.995510+0000","last_peered":"2023-10-21T05:18:22.995510+0000","last_clean":"2023-10-21T05:18:22.995510+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:18:22.995510+0000","last_undegraded":"2023-10-21T05:18:22.995510+0000","last_fullsized":"2023-10-21T05:18:22.995510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1619,"log_dups_size":0,"ondisk_log_size":1619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1636","reported_seq":2803,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:22.998136+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:18:22.998136+0000","last_peered":"2023-10-21T05:18:22.998136+0000","last_clean":"2023-10-21T05:18:22.998136+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:18:22.998136+0000","last_undegraded":"2023-10-21T05:18:22.998136+0000","last_fullsized":"2023-10-21T05:18:22.998136+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1636,"log_dups_size":0,"ondisk_log_size":1636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'242527","reported_seq":487127,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:25.627555+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:18:25.627555+0000","last_peered":"2023-10-21T05:18:25.627555+0000","last_clean":"2023-10-21T05:18:25.627555+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:18:25.627555+0000","last_undegraded":"2023-10-21T05:18:25.627555+0000","last_fullsized":"2023-10-21T05:18:25.627555+0000","mapping_epoch":17,"log_start":"54'232422","ondisk_log_start":"54'232422","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1189,"log_size":10105,"log_dups_size":0,"ondisk_log_size":10105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 421s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523079168,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":244314,"num_read_kb":950652,"num_write":242527,"num_write_kb":970108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1683","reported_seq":2738,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:18.803166+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:18:18.803166+0000","last_peered":"2023-10-21T05:18:18.803166+0000","last_clean":"2023-10-21T05:18:18.803166+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:18:18.803166+0000","last_undegraded":"2023-10-21T05:18:18.803166+0000","last_fullsized":"2023-10-21T05:18:18.803166+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1683,"log_dups_size":0,"ondisk_log_size":1683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'226420","reported_seq":455962,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:24.189081+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:18:24.189081+0000","last_peered":"2023-10-21T05:18:24.189081+0000","last_clean":"2023-10-21T05:18:24.189081+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:18:24.189081+0000","last_undegraded":"2023-10-21T05:18:24.189081+0000","last_fullsized":"2023-10-21T05:18:24.189081+0000","mapping_epoch":17,"log_start":"54'216413","ondisk_log_start":"54'216413","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190754304,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":229239,"num_read_kb":891568,"num_write":226420,"num_write_kb":905680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1055,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:16.060502+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:18:16.060502+0000","last_peered":"2023-10-21T05:18:16.060502+0000","last_clean":"2023-10-21T05:18:16.060502+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:18:16.060502+0000","last_undegraded":"2023-10-21T05:18:16.060502+0000","last_fullsized":"2023-10-21T05:18:16.060502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'235583","reported_seq":473881,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:25.623108+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:18:25.623108+0000","last_peered":"2023-10-21T05:18:25.623108+0000","last_clean":"2023-10-21T05:18:25.623108+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:18:25.623108+0000","last_undegraded":"2023-10-21T05:18:25.623108+0000","last_fullsized":"2023-10-21T05:18:25.623108+0000","mapping_epoch":17,"log_start":"54'225574","ondisk_log_start":"54'225574","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5385887744,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":237996,"num_read_kb":926632,"num_write":235583,"num_write_kb":942332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3902,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:18.793967+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:18:18.793967+0000","last_peered":"2023-10-21T05:18:18.793967+0000","last_clean":"2023-10-21T05:18:18.793967+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:18:18.793967+0000","last_undegraded":"2023-10-21T05:18:18.793967+0000","last_fullsized":"2023-10-21T05:18:18.793967+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'229963","reported_seq":463422,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:24.216745+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:18:24.216745+0000","last_peered":"2023-10-21T05:18:24.216745+0000","last_clean":"2023-10-21T05:18:24.216745+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:18:24.216745+0000","last_undegraded":"2023-10-21T05:18:24.216745+0000","last_fullsized":"2023-10-21T05:18:24.216745+0000","mapping_epoch":17,"log_start":"54'219886","ondisk_log_start":"54'219886","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":120,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 40s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261049875,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":233326,"num_read_kb":908004,"num_write":229963,"num_write_kb":919846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6296","reported_seq":10097,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:23.971283+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:18:23.971283+0000","last_peered":"2023-10-21T05:18:23.971283+0000","last_clean":"2023-10-21T05:18:23.971283+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:18:23.971283+0000","last_undegraded":"2023-10-21T05:18:23.971283+0000","last_fullsized":"2023-10-21T05:18:23.971283+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":227,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:02.445962+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:16:02.445962+0000","last_peered":"2023-10-21T05:16:02.445962+0000","last_clean":"2023-10-21T05:16:02.445962+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:16:02.445962+0000","last_undegraded":"2023-10-21T05:16:02.445962+0000","last_fullsized":"2023-10-21T05:16:02.445962+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21360771091,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":944875,"num_read_kb":3676856,"num_write":934493,"num_write_kb":3737966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32055320576,"data_stored":32055320576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40198,"ondisk_log_size":40198,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":24022,"ondisk_log_size":24022,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739246,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11835324,"kb_used_data":10435124,"kb_used_omap":8,"kb_used_meta":1400183,"kb_avail":82536516,"statfs":{"total":96636764160,"available":84517392384,"internally_reserved":0,"allocated":10685566976,"data_stored":10685262445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1433788061},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:43 2023","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":0.81799999999999995,"15min":0.93600000000000005},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16800000000000001},"max":{"1min":11.904999999999999,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":0.67500000000000004,"15min":0.79400000000000004},"min":{"1min":0.23499999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.252000000000001,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.46200000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:17:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.623},"min":{"1min":0.217,"5min":0.192,"15min":0.186},"max":{"1min":4.9939999999999998,"5min":4.9939999999999998,"15min":19.574000000000002},"last":0.5},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63,"15min":0.91900000000000004},"min":{"1min":0.252,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":1.252,"5min":5.8049999999999997,"15min":21.234999999999999},"last":0.52800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739245,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11861288,"kb_used_data":10435848,"kb_used_omap":15,"kb_used_meta":1425392,"kb_avail":82510552,"statfs":{"total":96636764160,"available":84490805248,"internally_reserved":0,"allocated":10686308352,"data_stored":10685890556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1459601798},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,15,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:27 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.86099999999999999,"15min":0.79700000000000004},"min":{"1min":0.17299999999999999,"5min":0.16300000000000001,"15min":0.13},"max":{"1min":3.524,"5min":14.682,"15min":14.682},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.43099999999999999,"15min":0.47099999999999997},"min":{"1min":0.23699999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.48299999999999998,"5min":2.9140000000000001,"15min":9.1210000000000004},"last":0.41699999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:17:27 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.56999999999999995,"15min":0.75900000000000001},"min":{"1min":0.25800000000000001,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":7.0789999999999997,"5min":7.0789999999999997,"15min":8.5389999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.61099999999999999,"15min":0.78100000000000003},"min":{"1min":0.33300000000000002,"5min":0.27800000000000002,"15min":0.20599999999999999},"max":{"1min":7.1130000000000004,"5min":7.1130000000000004,"15min":9.1950000000000003},"last":0.29499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739247,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11829024,"kb_used_data":10435800,"kb_used_omap":12,"kb_used_meta":1393203,"kb_avail":82542816,"statfs":{"total":96636764160,"available":84523843584,"internally_reserved":0,"allocated":10686259200,"data_stored":10685894203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1426640297},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,8,17,16,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.59799999999999998,"15min":0.58399999999999996},"min":{"1min":0.188,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":2.5369999999999999,"5min":5.6760000000000002,"15min":11.773},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":5.0609999999999999,"5min":5.7080000000000002,"15min":11.808999999999999},"last":0.38400000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:17:30 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.96099999999999997,"15min":0.77200000000000002},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":5.0800000000000001,"5min":5.6529999999999996,"15min":9.3140000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.999,"5min":0.85699999999999998,"15min":0.82199999999999995},"min":{"1min":0.28499999999999998,"5min":0.20899999999999999,"15min":0.17599999999999999},"max":{"1min":5.0469999999999997,"5min":5.0469999999999997,"15min":9.516},"last":0.41799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685120512,"data_stored":10685120512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685108224,"data_stored":10685108224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685091840,"data_stored":10685091840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:18:26.079 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:26.079 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:26.080 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:18:44.556 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:18:44.550+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:18:44.557 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:18:44.550+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:18:44.561 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:18:44.554+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:18:44.561 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:18:44.554+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:18:46.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:18:46.741 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:18:46.741 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:18:46.774 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2260,"stamp":"2023-10-21T05:18:45.741310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21361428323,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":953790,"num_read_kb":3694311,"num_write":938224,"num_write_kb":3753387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64369,"ondisk_log_size":64369,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35594488,"kb_used_data":31306908,"kb_used_omap":36,"kb_used_meta":4287515,"kb_avail":247521032,"statfs":{"total":289910292480,"available":253461536768,"internally_reserved":0,"allocated":32058273792,"data_stored":32057186468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":4390415820},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,3,10,13,16,13,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32768,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2234,"num_read_kb":8904,"num_write":2387,"num_write_kb":9548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002223"},"pg_stats":[{"pgid":"2.7","version":"54'4793","reported_seq":7704,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:44.393510+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:18:44.393510+0000","last_peered":"2023-10-21T05:18:44.393510+0000","last_clean":"2023-10-21T05:18:44.393510+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:18:44.393510+0000","last_undegraded":"2023-10-21T05:18:44.393510+0000","last_fullsized":"2023-10-21T05:18:44.393510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7941","reported_seq":12945,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:44.394249+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:18:44.394249+0000","last_peered":"2023-10-21T05:18:44.394249+0000","last_clean":"2023-10-21T05:18:44.394249+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:18:44.394249+0000","last_undegraded":"2023-10-21T05:18:44.394249+0000","last_fullsized":"2023-10-21T05:18:44.394249+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7941,"log_dups_size":0,"ondisk_log_size":7941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1624","reported_seq":2674,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:43.870551+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:18:43.870551+0000","last_peered":"2023-10-21T05:18:43.870551+0000","last_clean":"2023-10-21T05:18:43.870551+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:18:43.870551+0000","last_undegraded":"2023-10-21T05:18:43.870551+0000","last_fullsized":"2023-10-21T05:18:43.870551+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1640","reported_seq":2811,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:43.870607+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:18:43.870607+0000","last_peered":"2023-10-21T05:18:43.870607+0000","last_clean":"2023-10-21T05:18:43.870607+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:18:43.870607+0000","last_undegraded":"2023-10-21T05:18:43.870607+0000","last_fullsized":"2023-10-21T05:18:43.870607+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1640,"log_dups_size":0,"ondisk_log_size":1640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'243219","reported_seq":488550,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:40.824507+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:18:40.824507+0000","last_peered":"2023-10-21T05:18:40.824507+0000","last_clean":"2023-10-21T05:18:40.824507+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:18:40.824507+0000","last_undegraded":"2023-10-21T05:18:40.824507+0000","last_fullsized":"2023-10-21T05:18:40.824507+0000","mapping_epoch":17,"log_start":"54'233128","ondisk_log_start":"54'233128","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1230,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 436s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523079168,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":245037,"num_read_kb":953520,"num_write":243219,"num_write_kb":972876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1684","reported_seq":2743,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:39.277700+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:18:39.277700+0000","last_peered":"2023-10-21T05:18:39.277700+0000","last_clean":"2023-10-21T05:18:39.277700+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:18:39.277700+0000","last_undegraded":"2023-10-21T05:18:39.277700+0000","last_fullsized":"2023-10-21T05:18:39.277700+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1684,"log_dups_size":0,"ondisk_log_size":1684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'227387","reported_seq":457773,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:44.484093+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:18:44.484093+0000","last_peered":"2023-10-21T05:18:44.484093+0000","last_clean":"2023-10-21T05:18:44.484093+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:18:44.484093+0000","last_undegraded":"2023-10-21T05:18:44.484093+0000","last_fullsized":"2023-10-21T05:18:44.484093+0000","mapping_epoch":17,"log_start":"54'217332","ondisk_log_start":"54'217332","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190770688,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":230082,"num_read_kb":894936,"num_write":227387,"num_write_kb":909548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1059,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:34.167293+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:18:34.167293+0000","last_peered":"2023-10-21T05:18:34.167293+0000","last_clean":"2023-10-21T05:18:34.167293+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:18:34.167293+0000","last_undegraded":"2023-10-21T05:18:34.167293+0000","last_fullsized":"2023-10-21T05:18:34.167293+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'236314","reported_seq":475275,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:40.803161+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:18:40.803161+0000","last_peered":"2023-10-21T05:18:40.803161+0000","last_clean":"2023-10-21T05:18:40.803161+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:18:40.803161+0000","last_undegraded":"2023-10-21T05:18:40.803161+0000","last_fullsized":"2023-10-21T05:18:40.803161+0000","mapping_epoch":17,"log_start":"54'226283","ondisk_log_start":"54'226283","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5385904128,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":238657,"num_read_kb":929264,"num_write":236314,"num_write_kb":945256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3918,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:39.278651+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:18:39.278651+0000","last_peered":"2023-10-21T05:18:39.278651+0000","last_clean":"2023-10-21T05:18:39.278651+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:18:39.278651+0000","last_undegraded":"2023-10-21T05:18:39.278651+0000","last_fullsized":"2023-10-21T05:18:39.278651+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'230904","reported_seq":465291,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:44.488691+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:18:44.488691+0000","last_peered":"2023-10-21T05:18:44.488691+0000","last_clean":"2023-10-21T05:18:44.488691+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:18:44.488691+0000","last_undegraded":"2023-10-21T05:18:44.488691+0000","last_fullsized":"2023-10-21T05:18:44.488691+0000","mapping_epoch":17,"log_start":"54'220896","ondisk_log_start":"54'220896","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":161,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 60s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261082643,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":234248,"num_read_kb":911684,"num_write":230904,"num_write_kb":923610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6313","reported_seq":10130,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:44.404879+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:18:44.404879+0000","last_peered":"2023-10-21T05:18:44.404879+0000","last_clean":"2023-10-21T05:18:44.404879+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:18:44.404879+0000","last_undegraded":"2023-10-21T05:18:44.404879+0000","last_fullsized":"2023-10-21T05:18:44.404879+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":227,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:02.445962+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:16:02.445962+0000","last_peered":"2023-10-21T05:16:02.445962+0000","last_clean":"2023-10-21T05:16:02.445962+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:16:02.445962+0000","last_undegraded":"2023-10-21T05:16:02.445962+0000","last_fullsized":"2023-10-21T05:16:02.445962+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21360836627,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":948024,"num_read_kb":3689404,"num_write":937824,"num_write_kb":3751290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32055357440,"data_stored":32055357440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40185,"ondisk_log_size":40185,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":24085,"ondisk_log_size":24085,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739250,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11872196,"kb_used_data":10435196,"kb_used_omap":8,"kb_used_meta":1436983,"kb_avail":82499644,"statfs":{"total":96636764160,"available":84479635456,"internally_reserved":0,"allocated":10685640704,"data_stored":10685336173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1471471261},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:43 2023","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":0.81799999999999995,"15min":0.93600000000000005},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16800000000000001},"max":{"1min":11.904999999999999,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.2150000000000001,"5min":0.67500000000000004,"15min":0.79400000000000004},"min":{"1min":0.23499999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.252000000000001,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.93600000000000005}]},{"osd":1,"last update":"Sat Oct 21 05:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.56499999999999995,"15min":0.63300000000000001},"min":{"1min":0.21199999999999999,"5min":0.192,"15min":0.186},"max":{"1min":4.4690000000000003,"5min":4.9939999999999998,"15min":19.574000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.66000000000000003,"15min":0.92400000000000004},"min":{"1min":0.246,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":4.2089999999999996,"5min":5.8049999999999997,"15min":21.234999999999999},"last":0.58899999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739249,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11879336,"kb_used_data":10435872,"kb_used_omap":15,"kb_used_meta":1443440,"kb_avail":82492504,"statfs":{"total":96636764160,"available":84472324096,"internally_reserved":0,"allocated":10686332928,"data_stored":10685915132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1478082950},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,4,9,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:31 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":0.95299999999999996,"15min":0.85199999999999998},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.13},"max":{"1min":13.257,"5min":14.682,"15min":14.682},"last":5.3440000000000003},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.441,"15min":0.47399999999999998},"min":{"1min":0.19600000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.3679999999999999,"5min":2.9140000000000001,"15min":9.1210000000000004},"last":3.7890000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:18:31 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.56799999999999995,"15min":0.71999999999999997},"min":{"1min":0.23999999999999999,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":2.2909999999999999,"5min":7.0789999999999997,"15min":8.5389999999999997},"last":0.375},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.60899999999999999,"15min":0.74099999999999999},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.20599999999999999},"max":{"1min":2.3250000000000002,"5min":7.1130000000000004,"15min":9.1950000000000003},"last":0.40899999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739250,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11842956,"kb_used_data":10435840,"kb_used_omap":12,"kb_used_meta":1407091,"kb_avail":82528884,"statfs":{"total":96636764160,"available":84509577216,"internally_reserved":0,"allocated":10686300160,"data_stored":10685935163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1440861609},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,0,1,6,6,5,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:18:32 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.50600000000000001,"15min":0.55000000000000004},"min":{"1min":0.21099999999999999,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":1.7230000000000001,"5min":5.6760000000000002,"15min":6.5780000000000003},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.58199999999999996,"15min":0.61099999999999999},"min":{"1min":0.245,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":1.7929999999999999,"5min":5.7080000000000002,"15min":6.6429999999999998},"last":0.314}]},{"osd":2,"last update":"Sat Oct 21 05:18:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.84199999999999997,"15min":0.76700000000000002},"min":{"1min":0.28299999999999997,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.1400000000000001,"5min":5.6280000000000001,"15min":9.3140000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.78300000000000003,"15min":0.82499999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.17599999999999999},"max":{"1min":2.1549999999999998,"5min":5.0469999999999997,"15min":9.516},"last":0.36899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685140992,"data_stored":10685140992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685112320,"data_stored":10685112320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685104128,"data_stored":10685104128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:18:46.776 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:46.776 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:46.776 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:19:06.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:19:07.723 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:19:07.723 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:19:07.799 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2270,"stamp":"2023-10-21T05:19:05.744873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21361776483,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":957652,"num_read_kb":3709643,"num_write":942039,"num_write_kb":3768647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64488,"ondisk_log_size":64488,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35667188,"kb_used_data":31307420,"kb_used_omap":36,"kb_used_meta":4359707,"kb_avail":247448332,"statfs":{"total":289910292480,"available":253387091968,"internally_reserved":0,"allocated":32058798080,"data_stored":32057710756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":4464340428},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,9,30,5,16,19,18],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":212992,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2386,"num_read_kb":9492,"num_write":2380,"num_write_kb":9520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002100"},"pg_stats":[{"pgid":"2.7","version":"54'4824","reported_seq":7744,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.465174+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:19:04.465174+0000","last_peered":"2023-10-21T05:19:04.465174+0000","last_clean":"2023-10-21T05:19:04.465174+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:19:04.465174+0000","last_undegraded":"2023-10-21T05:19:04.465174+0000","last_fullsized":"2023-10-21T05:19:04.465174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4824,"log_dups_size":0,"ondisk_log_size":4824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'7992","reported_seq":13011,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.474911+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:19:04.474911+0000","last_peered":"2023-10-21T05:19:04.474911+0000","last_clean":"2023-10-21T05:19:04.474911+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:19:04.474911+0000","last_undegraded":"2023-10-21T05:19:04.474911+0000","last_fullsized":"2023-10-21T05:19:04.474911+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":7992,"log_dups_size":0,"ondisk_log_size":7992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1636","reported_seq":2690,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.443820+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:19:04.443820+0000","last_peered":"2023-10-21T05:19:04.443820+0000","last_clean":"2023-10-21T05:19:04.443820+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:19:04.443820+0000","last_undegraded":"2023-10-21T05:19:04.443820+0000","last_fullsized":"2023-10-21T05:19:04.443820+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1636,"log_dups_size":0,"ondisk_log_size":1636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1650","reported_seq":2825,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:03.949355+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:19:03.949355+0000","last_peered":"2023-10-21T05:19:03.949355+0000","last_clean":"2023-10-21T05:19:03.949355+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:19:03.949355+0000","last_undegraded":"2023-10-21T05:19:03.949355+0000","last_fullsized":"2023-10-21T05:19:03.949355+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1650,"log_dups_size":0,"ondisk_log_size":1650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'244246","reported_seq":490599,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:01.023625+0000","last_change":"2023-10-21T05:11:24.076961+0000","last_active":"2023-10-21T05:19:01.023625+0000","last_peered":"2023-10-21T05:19:01.023625+0000","last_clean":"2023-10-21T05:19:01.023625+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:19:01.023625+0000","last_undegraded":"2023-10-21T05:19:01.023625+0000","last_fullsized":"2023-10-21T05:19:01.023625+0000","mapping_epoch":17,"log_start":"54'234241","ondisk_log_start":"54'234241","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1280,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 456s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523079168,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":246049,"num_read_kb":957536,"num_write":244246,"num_write_kb":976984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1690","reported_seq":2753,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:59.797439+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:18:59.797439+0000","last_peered":"2023-10-21T05:18:59.797439+0000","last_clean":"2023-10-21T05:18:59.797439+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:18:59.797439+0000","last_undegraded":"2023-10-21T05:18:59.797439+0000","last_fullsized":"2023-10-21T05:18:59.797439+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1690,"log_dups_size":0,"ondisk_log_size":1690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'228271","reported_seq":459638,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.774358+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:19:04.774358+0000","last_peered":"2023-10-21T05:19:04.774358+0000","last_clean":"2023-10-21T05:19:04.774358+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:19:04.774358+0000","last_undegraded":"2023-10-21T05:19:04.774358+0000","last_fullsized":"2023-10-21T05:19:04.774358+0000","mapping_epoch":17,"log_start":"54'218246","ondisk_log_start":"54'218246","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190922240,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":231063,"num_read_kb":898840,"num_write":228271,"num_write_kb":913084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1063,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:54.641270+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:18:54.641270+0000","last_peered":"2023-10-21T05:18:54.641270+0000","last_clean":"2023-10-21T05:18:54.641270+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:18:54.641270+0000","last_undegraded":"2023-10-21T05:18:54.641270+0000","last_fullsized":"2023-10-21T05:18:54.641270+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'237259","reported_seq":477163,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:01.020638+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:19:01.020638+0000","last_peered":"2023-10-21T05:19:01.020638+0000","last_clean":"2023-10-21T05:19:01.020638+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:19:01.020638+0000","last_undegraded":"2023-10-21T05:19:01.020638+0000","last_fullsized":"2023-10-21T05:19:01.020638+0000","mapping_epoch":17,"log_start":"54'227186","ondisk_log_start":"54'227186","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5385949184,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":239600,"num_read_kb":932996,"num_write":237259,"num_write_kb":949036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3934,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:59.777215+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:18:59.777215+0000","last_peered":"2023-10-21T05:18:59.777215+0000","last_clean":"2023-10-21T05:18:59.777215+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:18:59.777215+0000","last_undegraded":"2023-10-21T05:18:59.777215+0000","last_fullsized":"2023-10-21T05:18:59.777215+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'231863","reported_seq":467184,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:04.787275+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:19:04.787275+0000","last_peered":"2023-10-21T05:19:04.787275+0000","last_clean":"2023-10-21T05:19:04.787275+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:19:04.787275+0000","last_undegraded":"2023-10-21T05:19:04.787275+0000","last_fullsized":"2023-10-21T05:19:04.787275+0000","mapping_epoch":17,"log_start":"54'221812","ondisk_log_start":"54'221812","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":218,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 80s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261234195,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":235174,"num_read_kb":915364,"num_write":231863,"num_write_kb":927446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6353","reported_seq":10183,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.505784+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:19:04.505784+0000","last_peered":"2023-10-21T05:19:04.505784+0000","last_clean":"2023-10-21T05:19:04.505784+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:19:04.505784+0000","last_undegraded":"2023-10-21T05:19:04.505784+0000","last_fullsized":"2023-10-21T05:19:04.505784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6353,"log_dups_size":0,"ondisk_log_size":6353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":227,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:02.445962+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:16:02.445962+0000","last_peered":"2023-10-21T05:16:02.445962+0000","last_clean":"2023-10-21T05:16:02.445962+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:16:02.445962+0000","last_undegraded":"2023-10-21T05:16:02.445962+0000","last_fullsized":"2023-10-21T05:16:02.445962+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21361184787,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":951886,"num_read_kb":3704736,"num_write":941639,"num_write_kb":3766550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32055918592,"data_stored":32055918592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40154,"ondisk_log_size":40154,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":24235,"ondisk_log_size":24235,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739254,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11890428,"kb_used_data":10435380,"kb_used_omap":8,"kb_used_meta":1455031,"kb_avail":82481412,"statfs":{"total":96636764160,"available":84460965888,"internally_reserved":0,"allocated":10685829120,"data_stored":10685524589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1489952413},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,7,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.82399999999999995,"15min":0.90100000000000002},"min":{"1min":0.183,"5min":0.17299999999999999,"15min":0.16800000000000001},"max":{"1min":4.3650000000000002,"5min":11.904999999999999,"15min":22.969000000000001},"last":1.133},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.69099999999999995,"15min":0.78200000000000003},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.93600000000000005,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.93300000000000005}]},{"osd":1,"last update":"Sat Oct 21 05:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.56499999999999995,"15min":0.63300000000000001},"min":{"1min":0.21199999999999999,"5min":0.192,"15min":0.186},"max":{"1min":4.4690000000000003,"5min":4.9939999999999998,"15min":19.574000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.66000000000000003,"15min":0.92400000000000004},"min":{"1min":0.246,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":4.2089999999999996,"5min":5.8049999999999997,"15min":21.234999999999999},"last":0.23699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739253,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11897572,"kb_used_data":10436060,"kb_used_omap":15,"kb_used_meta":1461488,"kb_avail":82474268,"statfs":{"total":96636764160,"available":84453650432,"internally_reserved":0,"allocated":10686525440,"data_stored":10686107644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1496564102},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,1,6,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:31 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":0.95299999999999996,"15min":0.85199999999999998},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.13},"max":{"1min":13.257,"5min":14.682,"15min":14.682},"last":2.468},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.441,"15min":0.47399999999999998},"min":{"1min":0.19600000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.3679999999999999,"5min":2.9140000000000001,"15min":9.1210000000000004},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:18:31 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.56799999999999995,"15min":0.71999999999999997},"min":{"1min":0.23999999999999999,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":2.2909999999999999,"5min":7.0789999999999997,"15min":8.5389999999999997},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.60899999999999999,"15min":0.74099999999999999},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.20599999999999999},"max":{"1min":2.3250000000000002,"5min":7.1130000000000004,"15min":9.1950000000000003},"last":0.44800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739254,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11879188,"kb_used_data":10435980,"kb_used_omap":12,"kb_used_meta":1443187,"kb_avail":82492652,"statfs":{"total":96636764160,"available":84472475648,"internally_reserved":0,"allocated":10686443520,"data_stored":10686078523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1477823913},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,16,3,8,18,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:18:32 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.50600000000000001,"15min":0.55000000000000004},"min":{"1min":0.21099999999999999,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":1.7230000000000001,"5min":5.6760000000000002,"15min":6.5780000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.58199999999999996,"15min":0.61099999999999999},"min":{"1min":0.245,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":1.7929999999999999,"5min":5.7080000000000002,"15min":6.6429999999999998},"last":0.44400000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:18:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.84199999999999997,"15min":0.76700000000000002},"min":{"1min":0.28299999999999997,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.1400000000000001,"5min":5.6280000000000001,"15min":9.3140000000000001},"last":3.323},{"interface":"front","average":{"1min":0.434,"5min":0.78300000000000003,"15min":0.82499999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.17599999999999999},"max":{"1min":2.1549999999999998,"5min":5.0469999999999997,"15min":9.516},"last":3.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685284352,"data_stored":10685284352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685325312,"data_stored":10685325312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685308928,"data_stored":10685308928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:19:07.801 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:07.801 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:07.802 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:19:27.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:19:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:19:28.365 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:19:28.389 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2281,"stamp":"2023-10-21T05:19:27.749712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21362108259,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":961861,"num_read_kb":3726359,"num_write":946311,"num_write_kb":3785735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64617,"ondisk_log_size":64617,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35748208,"kb_used_data":31307928,"kb_used_omap":36,"kb_used_meta":4440219,"kb_avail":247367312,"statfs":{"total":289910292480,"available":253304127488,"internally_reserved":0,"allocated":32059318272,"data_stored":32058230948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":4546784716},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,1,3,4,7,15,25,11,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":225280,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2298,"num_read_kb":9128,"num_write":2385,"num_write_kb":9540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003121"},"pg_stats":[{"pgid":"2.7","version":"54'4848","reported_seq":7780,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:21.286545+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:19:21.286545+0000","last_peered":"2023-10-21T05:19:21.286545+0000","last_clean":"2023-10-21T05:19:21.286545+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:19:21.286545+0000","last_undegraded":"2023-10-21T05:19:21.286545+0000","last_fullsized":"2023-10-21T05:19:21.286545+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4848,"log_dups_size":0,"ondisk_log_size":4848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8032","reported_seq":13071,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:21.298601+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:19:21.298601+0000","last_peered":"2023-10-21T05:19:21.298601+0000","last_clean":"2023-10-21T05:19:21.298601+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:19:21.298601+0000","last_undegraded":"2023-10-21T05:19:21.298601+0000","last_fullsized":"2023-10-21T05:19:21.298601+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8032,"log_dups_size":0,"ondisk_log_size":8032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1644","reported_seq":2702,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:20.233735+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:19:20.233735+0000","last_peered":"2023-10-21T05:19:20.233735+0000","last_clean":"2023-10-21T05:19:20.233735+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:19:20.233735+0000","last_undegraded":"2023-10-21T05:19:20.233735+0000","last_fullsized":"2023-10-21T05:19:20.233735+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1644,"log_dups_size":0,"ondisk_log_size":1644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1658","reported_seq":2837,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:20.233900+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:19:20.233900+0000","last_peered":"2023-10-21T05:19:20.233900+0000","last_clean":"2023-10-21T05:19:20.233900+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:19:20.233900+0000","last_undegraded":"2023-10-21T05:19:20.233900+0000","last_fullsized":"2023-10-21T05:19:20.233900+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1658,"log_dups_size":0,"ondisk_log_size":1658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'245495","reported_seq":493096,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:26.177236+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:19:26.177236+0000","last_peered":"2023-10-21T05:19:26.177236+0000","last_clean":"2023-10-21T05:19:26.177236+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:19:26.177236+0000","last_undegraded":"2023-10-21T05:19:26.177236+0000","last_fullsized":"2023-10-21T05:19:26.177236+0000","mapping_epoch":17,"log_start":"54'235464","ondisk_log_start":"54'235464","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523169280,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":247281,"num_read_kb":962424,"num_write":245495,"num_write_kb":981980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1697","reported_seq":2765,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:25.358384+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:19:25.358384+0000","last_peered":"2023-10-21T05:19:25.358384+0000","last_clean":"2023-10-21T05:19:25.358384+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:19:25.358384+0000","last_undegraded":"2023-10-21T05:19:25.358384+0000","last_fullsized":"2023-10-21T05:19:25.358384+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1697,"log_dups_size":0,"ondisk_log_size":1697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'229201","reported_seq":461458,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:24.907079+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:19:24.907079+0000","last_peered":"2023-10-21T05:19:24.907079+0000","last_clean":"2023-10-21T05:19:24.907079+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:19:24.907079+0000","last_undegraded":"2023-10-21T05:19:24.907079+0000","last_fullsized":"2023-10-21T05:19:24.907079+0000","mapping_epoch":17,"log_start":"54'219153","ondisk_log_start":"54'219153","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190922240,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":231954,"num_read_kb":902384,"num_write":229201,"num_write_kb":916804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1068,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:20.233655+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:19:20.233655+0000","last_peered":"2023-10-21T05:19:20.233655+0000","last_clean":"2023-10-21T05:19:20.233655+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:19:20.233655+0000","last_undegraded":"2023-10-21T05:19:20.233655+0000","last_fullsized":"2023-10-21T05:19:20.233655+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'238431","reported_seq":479481,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:26.207675+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:19:26.207675+0000","last_peered":"2023-10-21T05:19:26.207675+0000","last_clean":"2023-10-21T05:19:26.207675+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:19:26.207675+0000","last_undegraded":"2023-10-21T05:19:26.207675+0000","last_fullsized":"2023-10-21T05:19:26.207675+0000","mapping_epoch":17,"log_start":"54'228404","ondisk_log_start":"54'228404","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386084352,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":240748,"num_read_kb":937552,"num_write":238431,"num_write_kb":953724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3954,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:25.441412+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:19:25.441412+0000","last_peered":"2023-10-21T05:19:25.441412+0000","last_clean":"2023-10-21T05:19:25.441412+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:19:25.441412+0000","last_undegraded":"2023-10-21T05:19:25.441412+0000","last_fullsized":"2023-10-21T05:19:25.441412+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'232784","reported_seq":469054,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:24.905370+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:19:24.905370+0000","last_peered":"2023-10-21T05:19:24.905370+0000","last_clean":"2023-10-21T05:19:24.905370+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:19:24.905370+0000","last_undegraded":"2023-10-21T05:19:24.905370+0000","last_fullsized":"2023-10-21T05:19:24.905370+0000","mapping_epoch":17,"log_start":"54'222726","ondisk_log_start":"54'222726","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":272,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 100s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261340691,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":236112,"num_read_kb":919092,"num_write":232784,"num_write_kb":931130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6385","reported_seq":10227,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:21.363622+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:19:21.363622+0000","last_peered":"2023-10-21T05:19:21.363622+0000","last_clean":"2023-10-21T05:19:21.363622+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:19:21.363622+0000","last_undegraded":"2023-10-21T05:19:21.363622+0000","last_fullsized":"2023-10-21T05:19:21.363622+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6385,"log_dups_size":0,"ondisk_log_size":6385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":227,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:02.445962+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:16:02.445962+0000","last_peered":"2023-10-21T05:16:02.445962+0000","last_clean":"2023-10-21T05:16:02.445962+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:16:02.445962+0000","last_undegraded":"2023-10-21T05:16:02.445962+0000","last_fullsized":"2023-10-21T05:16:02.445962+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21361516563,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":956095,"num_read_kb":3721452,"num_write":945911,"num_write_kb":3783638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32056467456,"data_stored":32056467456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40164,"ondisk_log_size":40164,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":24354,"ondisk_log_size":24354,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739259,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11930820,"kb_used_data":10435516,"kb_used_omap":8,"kb_used_meta":1495287,"kb_avail":82441020,"statfs":{"total":96636764160,"available":84419604480,"internally_reserved":0,"allocated":10685968384,"data_stored":10685663853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1531174557},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.82399999999999995,"15min":0.90100000000000002},"min":{"1min":0.183,"5min":0.17299999999999999,"15min":0.16800000000000001},"max":{"1min":4.3650000000000002,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.69099999999999995,"15min":0.78200000000000003},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.93600000000000005,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.32800000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.56499999999999995,"15min":0.63300000000000001},"min":{"1min":0.21199999999999999,"5min":0.192,"15min":0.186},"max":{"1min":4.4690000000000003,"5min":4.9939999999999998,"15min":19.574000000000002},"last":0.308},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.66000000000000003,"15min":0.92400000000000004},"min":{"1min":0.246,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":4.2089999999999996,"5min":5.8049999999999997,"15min":21.234999999999999},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739257,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11915788,"kb_used_data":10436228,"kb_used_omap":15,"kb_used_meta":1479536,"kb_avail":82456052,"statfs":{"total":96636764160,"available":84434997248,"internally_reserved":0,"allocated":10686697472,"data_stored":10686279676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1515045254},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,5,7,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:31 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":0.95299999999999996,"15min":0.85199999999999998},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.13},"max":{"1min":13.257,"5min":14.682,"15min":14.682},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.441,"15min":0.47399999999999998},"min":{"1min":0.19600000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.3679999999999999,"5min":2.9140000000000001,"15min":9.1210000000000004},"last":4.0960000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:18:31 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.56799999999999995,"15min":0.71999999999999997},"min":{"1min":0.23999999999999999,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":2.2909999999999999,"5min":7.0789999999999997,"15min":8.5389999999999997},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.60899999999999999,"15min":0.74099999999999999},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.20599999999999999},"max":{"1min":2.3250000000000002,"5min":7.1130000000000004,"15min":9.1950000000000003},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739259,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11901600,"kb_used_data":10436184,"kb_used_omap":12,"kb_used_meta":1465395,"kb_avail":82470240,"statfs":{"total":96636764160,"available":84449525760,"internally_reserved":0,"allocated":10686652416,"data_stored":10686287419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1500564905},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,1,3,3,7,13,7,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:18:32 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.50600000000000001,"15min":0.55000000000000004},"min":{"1min":0.21099999999999999,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":1.7230000000000001,"5min":5.6760000000000002,"15min":6.5780000000000003},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.58199999999999996,"15min":0.61099999999999999},"min":{"1min":0.245,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":1.7929999999999999,"5min":5.7080000000000002,"15min":6.6429999999999998},"last":0.32000000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:18:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.84199999999999997,"15min":0.76700000000000002},"min":{"1min":0.28299999999999997,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.1400000000000001,"5min":5.6280000000000001,"15min":9.3140000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.434,"5min":0.78300000000000003,"15min":0.82499999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.17599999999999999},"max":{"1min":2.1549999999999998,"5min":5.0469999999999997,"15min":9.516},"last":0.19700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685505536,"data_stored":10685505536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685480960,"data_stored":10685480960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685480960,"data_stored":10685480960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:19:28.391 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:28.391 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:19:48.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:19:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:19:49.168 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:19:49.214 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2291,"stamp":"2023-10-21T05:19:47.753280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21362276195,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":965782,"num_read_kb":3741971,"num_write":950268,"num_write_kb":3801563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64708,"ondisk_log_size":64708,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35838812,"kb_used_data":31308280,"kb_used_omap":36,"kb_used_meta":4530459,"kb_avail":247276708,"statfs":{"total":289910292480,"available":253211348992,"internally_reserved":0,"allocated":32059678720,"data_stored":32058591396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":4639190476},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,5,14,16,13,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":110592,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2397,"num_read_kb":9540,"num_write":2493,"num_write_kb":9972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002147"},"pg_stats":[{"pgid":"2.7","version":"54'4866","reported_seq":7811,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:40.717637+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:19:40.717637+0000","last_peered":"2023-10-21T05:19:40.717637+0000","last_clean":"2023-10-21T05:19:40.717637+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:19:40.717637+0000","last_undegraded":"2023-10-21T05:19:40.717637+0000","last_fullsized":"2023-10-21T05:19:40.717637+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8062","reported_seq":13122,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:40.718396+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:19:40.718396+0000","last_peered":"2023-10-21T05:19:40.718396+0000","last_clean":"2023-10-21T05:19:40.718396+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:19:40.718396+0000","last_undegraded":"2023-10-21T05:19:40.718396+0000","last_fullsized":"2023-10-21T05:19:40.718396+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8062,"log_dups_size":0,"ondisk_log_size":8062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1650","reported_seq":2712,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:39.124047+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:19:39.124047+0000","last_peered":"2023-10-21T05:19:39.124047+0000","last_clean":"2023-10-21T05:19:39.124047+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:19:39.124047+0000","last_undegraded":"2023-10-21T05:19:39.124047+0000","last_fullsized":"2023-10-21T05:19:39.124047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1650,"log_dups_size":0,"ondisk_log_size":1650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1664","reported_seq":2847,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:39.123932+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:19:39.123932+0000","last_peered":"2023-10-21T05:19:39.123932+0000","last_clean":"2023-10-21T05:19:39.123932+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:19:39.123932+0000","last_undegraded":"2023-10-21T05:19:39.123932+0000","last_fullsized":"2023-10-21T05:19:39.123932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1664,"log_dups_size":0,"ondisk_log_size":1664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'246537","reported_seq":495162,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:46.557448+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:19:46.557448+0000","last_peered":"2023-10-21T05:19:46.557448+0000","last_clean":"2023-10-21T05:19:46.557448+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:19:46.557448+0000","last_undegraded":"2023-10-21T05:19:46.557448+0000","last_fullsized":"2023-10-21T05:19:46.557448+0000","mapping_epoch":17,"log_start":"54'236484","ondisk_log_start":"54'236484","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523226624,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":248305,"num_read_kb":966500,"num_write":246537,"num_write_kb":986148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1703","reported_seq":2775,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:45.142877+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:19:45.142877+0000","last_peered":"2023-10-21T05:19:45.142877+0000","last_clean":"2023-10-21T05:19:45.142877+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:19:45.142877+0000","last_undegraded":"2023-10-21T05:19:45.142877+0000","last_fullsized":"2023-10-21T05:19:45.142877+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1703,"log_dups_size":0,"ondisk_log_size":1703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'230172","reported_seq":463398,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:45.069957+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:19:45.069957+0000","last_peered":"2023-10-21T05:19:45.069957+0000","last_clean":"2023-10-21T05:19:45.069957+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:19:45.069957+0000","last_undegraded":"2023-10-21T05:19:45.069957+0000","last_fullsized":"2023-10-21T05:19:45.069957+0000","mapping_epoch":17,"log_start":"54'220164","ondisk_log_start":"54'220164","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190975488,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":232922,"num_read_kb":906236,"num_write":230172,"num_write_kb":920688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1072,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:40.734212+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:19:40.734212+0000","last_peered":"2023-10-21T05:19:40.734212+0000","last_clean":"2023-10-21T05:19:40.734212+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:19:40.734212+0000","last_undegraded":"2023-10-21T05:19:40.734212+0000","last_fullsized":"2023-10-21T05:19:40.734212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'239406","reported_seq":481401,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:46.521652+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:19:46.521652+0000","last_peered":"2023-10-21T05:19:46.521652+0000","last_clean":"2023-10-21T05:19:46.521652+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:19:46.521652+0000","last_undegraded":"2023-10-21T05:19:46.521652+0000","last_fullsized":"2023-10-21T05:19:46.521652+0000","mapping_epoch":17,"log_start":"54'229309","ondisk_log_start":"54'229309","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386133504,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":241701,"num_read_kb":941344,"num_write":239406,"num_write_kb":957624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3970,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:45.834164+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:19:45.834164+0000","last_peered":"2023-10-21T05:19:45.834164+0000","last_clean":"2023-10-21T05:19:45.834164+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:19:45.834164+0000","last_undegraded":"2023-10-21T05:19:45.834164+0000","last_fullsized":"2023-10-21T05:19:45.834164+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'233753","reported_seq":471009,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:45.078431+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:19:45.078431+0000","last_peered":"2023-10-21T05:19:45.078431+0000","last_clean":"2023-10-21T05:19:45.078431+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:19:45.078431+0000","last_undegraded":"2023-10-21T05:19:45.078431+0000","last_fullsized":"2023-10-21T05:19:45.078431+0000","mapping_epoch":17,"log_start":"54'223746","ondisk_log_start":"54'223746","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":346,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 120s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261348883,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":237088,"num_read_kb":922984,"num_write":233753,"num_write_kb":935006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6409","reported_seq":10267,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:19:40.719829+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:19:40.719829+0000","last_peered":"2023-10-21T05:19:40.719829+0000","last_clean":"2023-10-21T05:19:40.719829+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:19:40.719829+0000","last_undegraded":"2023-10-21T05:19:40.719829+0000","last_fullsized":"2023-10-21T05:19:40.719829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6409,"log_dups_size":0,"ondisk_log_size":6409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":227,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:16:02.445962+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:16:02.445962+0000","last_peered":"2023-10-21T05:16:02.445962+0000","last_clean":"2023-10-21T05:16:02.445962+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:16:02.445962+0000","last_undegraded":"2023-10-21T05:16:02.445962+0000","last_fullsized":"2023-10-21T05:16:02.445962+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21361684499,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":960016,"num_read_kb":3737064,"num_write":949868,"num_write_kb":3799466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32056713216,"data_stored":32056713216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40165,"ondisk_log_size":40165,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":24444,"ondisk_log_size":24444,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739263,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11949012,"kb_used_data":10435652,"kb_used_omap":8,"kb_used_meta":1513335,"kb_avail":82422828,"statfs":{"total":96636764160,"available":84400975872,"internally_reserved":0,"allocated":10686107648,"data_stored":10685803117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1549655709},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.82399999999999995,"15min":0.90100000000000002},"min":{"1min":0.183,"5min":0.17299999999999999,"15min":0.16800000000000001},"max":{"1min":4.3650000000000002,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.69099999999999995,"15min":0.78200000000000003},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.93600000000000005,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.27800000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:19:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.58899999999999997,"15min":0.63100000000000001},"min":{"1min":0.22900000000000001,"5min":0.19700000000000001,"15min":0.186},"max":{"1min":4.6760000000000002,"5min":4.9939999999999998,"15min":19.574000000000002},"last":4.1269999999999998},{"interface":"front","average":{"1min":1.6379999999999999,"5min":0.88400000000000001,"15min":0.94699999999999995},"min":{"1min":0.23699999999999999,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.308}]}]},{"osd":1,"up_from":8,"seq":34359739261,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11951996,"kb_used_data":10436340,"kb_used_omap":15,"kb_used_meta":1515632,"kb_avail":82419844,"statfs":{"total":96636764160,"available":84397920256,"internally_reserved":0,"allocated":10686812160,"data_stored":10686394364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1552007558},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,8,9,7,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":0.90200000000000002,"15min":0.82599999999999996},"min":{"1min":0.191,"5min":0.16900000000000001,"15min":0.13},"max":{"1min":6.5209999999999999,"5min":13.257,"15min":14.682},"last":0.377},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.51700000000000002,"15min":0.505},"min":{"1min":0.218,"5min":0.19600000000000001,"15min":0.18099999999999999},"max":{"1min":4.0960000000000001,"5min":4.0960000000000001,"15min":9.1210000000000004},"last":1.425}]},{"osd":2,"last update":"Sat Oct 21 05:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53900000000000003,"15min":0.72599999999999998},"min":{"1min":0.28699999999999998,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":1.3700000000000001,"5min":7.0789999999999997,"15min":8.5389999999999997},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.58199999999999996,"15min":0.747},"min":{"1min":0.34599999999999997,"5min":0.27900000000000003,"15min":0.20599999999999999},"max":{"1min":1.409,"5min":7.1130000000000004,"15min":9.1950000000000003},"last":0.35799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739263,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11937804,"kb_used_data":10436288,"kb_used_omap":12,"kb_used_meta":1501491,"kb_avail":82434036,"statfs":{"total":96636764160,"available":84412452864,"internally_reserved":0,"allocated":10686758912,"data_stored":10686393915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1537527209},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,7,5,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:19:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":5.3330000000000002,"5min":5.6760000000000002,"15min":6.5780000000000003},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.71699999999999997,"15min":0.65800000000000003},"min":{"1min":0.26600000000000001,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":5.4690000000000003,"5min":5.7080000000000002,"15min":6.6429999999999998},"last":1.552}]},{"osd":2,"last update":"Sat Oct 21 05:19:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":0.95199999999999996,"15min":0.80500000000000005},"min":{"1min":0.28399999999999997,"5min":0.26100000000000001,"15min":0.20899999999999999},"max":{"1min":5.4989999999999997,"5min":5.6280000000000001,"15min":9.3140000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.84799999999999998,"15min":0.85199999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":5.3979999999999997,"5min":5.3979999999999997,"15min":9.516},"last":0.28199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685599744,"data_stored":10685599744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685530112,"data_stored":10685530112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685583360,"data_stored":10685583360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:19:49.216 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:49.217 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:20:00.291 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T05:20:00.292 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:20:00.292 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T05:20:00.292 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:20:00.292 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T05:20:00.293 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:20:00.293 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:20:00.293 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:20:00.294 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:20:00.294 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T05:20:00.295 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T05:20:00.295 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T05:20:00.295 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:20:00.296 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:20:00.288+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:20:09.218 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:20:10.160 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:20:10.161 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:20:10.201 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2302,"stamp":"2023-10-21T05:20:09.757174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21362386787,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":969897,"num_read_kb":3758363,"num_write":954165,"num_write_kb":3817151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64848,"ondisk_log_size":64848,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35929196,"kb_used_data":31308428,"kb_used_omap":36,"kb_used_meta":4620699,"kb_avail":247186324,"statfs":{"total":289910292480,"available":253118795776,"internally_reserved":0,"allocated":32059830272,"data_stored":32058742948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":4731596236},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,9,13,15,17,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":61440,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2115,"num_read_kb":8432,"num_write":1968,"num_write_kb":7872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002052"},"pg_stats":[{"pgid":"2.7","version":"54'4881","reported_seq":7838,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:01.193529+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:20:01.193529+0000","last_peered":"2023-10-21T05:20:01.193529+0000","last_clean":"2023-10-21T05:20:01.193529+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:20:01.193529+0000","last_undegraded":"2023-10-21T05:20:01.193529+0000","last_fullsized":"2023-10-21T05:20:01.193529+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8087","reported_seq":13167,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:01.194379+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:20:01.194379+0000","last_peered":"2023-10-21T05:20:01.194379+0000","last_clean":"2023-10-21T05:20:01.194379+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:20:01.194379+0000","last_undegraded":"2023-10-21T05:20:01.194379+0000","last_fullsized":"2023-10-21T05:20:01.194379+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8087,"log_dups_size":0,"ondisk_log_size":8087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1655","reported_seq":2721,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:00.578404+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:20:00.578404+0000","last_peered":"2023-10-21T05:20:00.578404+0000","last_clean":"2023-10-21T05:20:00.578404+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:20:00.578404+0000","last_undegraded":"2023-10-21T05:20:00.578404+0000","last_fullsized":"2023-10-21T05:20:00.578404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1655,"log_dups_size":0,"ondisk_log_size":1655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1669","reported_seq":2856,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:00.578453+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:20:00.578453+0000","last_peered":"2023-10-21T05:20:00.578453+0000","last_clean":"2023-10-21T05:20:00.578453+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:20:00.578453+0000","last_undegraded":"2023-10-21T05:20:00.578453+0000","last_fullsized":"2023-10-21T05:20:00.578453+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1669,"log_dups_size":0,"ondisk_log_size":1669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'247507","reported_seq":497174,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:06.745617+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:20:06.745617+0000","last_peered":"2023-10-21T05:20:06.745617+0000","last_clean":"2023-10-21T05:20:06.745617+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:20:06.745617+0000","last_undegraded":"2023-10-21T05:20:06.745617+0000","last_fullsized":"2023-10-21T05:20:06.745617+0000","mapping_epoch":17,"log_start":"54'237491","ondisk_log_start":"54'237491","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523234816,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":249348,"num_read_kb":970644,"num_write":247507,"num_write_kb":990028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1709","reported_seq":2785,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:01.197180+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:20:01.197180+0000","last_peered":"2023-10-21T05:20:01.197180+0000","last_clean":"2023-10-21T05:20:01.197180+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:20:01.197180+0000","last_undegraded":"2023-10-21T05:20:01.197180+0000","last_fullsized":"2023-10-21T05:20:01.197180+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1709,"log_dups_size":0,"ondisk_log_size":1709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'231136","reported_seq":465355,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:05.367206+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:20:05.367206+0000","last_peered":"2023-10-21T05:20:05.367206+0000","last_clean":"2023-10-21T05:20:05.367206+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:20:05.367206+0000","last_undegraded":"2023-10-21T05:20:05.367206+0000","last_fullsized":"2023-10-21T05:20:05.367206+0000","mapping_epoch":17,"log_start":"54'221071","ondisk_log_start":"54'221071","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190979584,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":233916,"num_read_kb":910208,"num_write":231136,"num_write_kb":924544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1076,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:01.193686+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:20:01.193686+0000","last_peered":"2023-10-21T05:20:01.193686+0000","last_clean":"2023-10-21T05:20:01.193686+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:20:01.193686+0000","last_undegraded":"2023-10-21T05:20:01.193686+0000","last_fullsized":"2023-10-21T05:20:01.193686+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'240396","reported_seq":483457,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:06.751097+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:20:06.751097+0000","last_peered":"2023-10-21T05:20:06.751097+0000","last_clean":"2023-10-21T05:20:06.751097+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:20:06.751097+0000","last_undegraded":"2023-10-21T05:20:06.751097+0000","last_fullsized":"2023-10-21T05:20:06.751097+0000","mapping_epoch":17,"log_start":"54'230321","ondisk_log_start":"54'230321","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386178560,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":242758,"num_read_kb":945556,"num_write":240396,"num_write_kb":961584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3987,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:06.317804+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:20:06.317804+0000","last_peered":"2023-10-21T05:20:06.317804+0000","last_clean":"2023-10-21T05:20:06.317804+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:20:06.317804+0000","last_undegraded":"2023-10-21T05:20:06.317804+0000","last_fullsized":"2023-10-21T05:20:06.317804+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'234726","reported_seq":473013,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:05.325599+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:20:05.325599+0000","last_peered":"2023-10-21T05:20:05.325599+0000","last_clean":"2023-10-21T05:20:05.325599+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:20:05.325599+0000","last_undegraded":"2023-10-21T05:20:05.325599+0000","last_fullsized":"2023-10-21T05:20:05.325599+0000","mapping_epoch":17,"log_start":"54'224653","ondisk_log_start":"54'224653","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":421,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 141s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261402131,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":238109,"num_read_kb":927048,"num_write":234726,"num_write_kb":938898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6429","reported_seq":10303,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:01.195150+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:20:01.195150+0000","last_peered":"2023-10-21T05:20:01.195150+0000","last_clean":"2023-10-21T05:20:01.195150+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:20:01.195150+0000","last_undegraded":"2023-10-21T05:20:01.195150+0000","last_fullsized":"2023-10-21T05:20:01.195150+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6429,"log_dups_size":0,"ondisk_log_size":6429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":228,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:04.051678+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:18:04.051678+0000","last_peered":"2023-10-21T05:18:04.051678+0000","last_clean":"2023-10-21T05:18:04.051678+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:18:04.051678+0000","last_undegraded":"2023-10-21T05:18:04.051678+0000","last_fullsized":"2023-10-21T05:18:04.051678+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21361795091,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":964131,"num_read_kb":3753456,"num_write":953765,"num_write_kb":3815054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32056967168,"data_stored":32056967168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40229,"ondisk_log_size":40229,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":24520,"ondisk_log_size":24520,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739267,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11967100,"kb_used_data":10435700,"kb_used_omap":8,"kb_used_meta":1531383,"kb_avail":82404740,"statfs":{"total":96636764160,"available":84382453760,"internally_reserved":0,"allocated":10686156800,"data_stored":10685852269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1568136861},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":0.96399999999999997,"15min":0.876},"min":{"1min":0.19,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":6.0270000000000001,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.056,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.20399999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.8689999999999998,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.25800000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:19:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.58899999999999997,"15min":0.63100000000000001},"min":{"1min":0.22900000000000001,"5min":0.19700000000000001,"15min":0.186},"max":{"1min":4.6760000000000002,"5min":4.9939999999999998,"15min":19.574000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.6379999999999999,"5min":0.88400000000000001,"15min":0.94699999999999995},"min":{"1min":0.23699999999999999,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.224}]}]},{"osd":1,"up_from":8,"seq":34359739265,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11988112,"kb_used_data":10436360,"kb_used_omap":15,"kb_used_meta":1551728,"kb_avail":82383728,"statfs":{"total":96636764160,"available":84360937472,"internally_reserved":0,"allocated":10686832640,"data_stored":10686414844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1588969862},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,11,7,13,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":0.90200000000000002,"15min":0.82599999999999996},"min":{"1min":0.191,"5min":0.16900000000000001,"15min":0.13},"max":{"1min":6.5209999999999999,"5min":13.257,"15min":14.682},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.51700000000000002,"15min":0.505},"min":{"1min":0.218,"5min":0.19600000000000001,"15min":0.18099999999999999},"max":{"1min":4.0960000000000001,"5min":4.0960000000000001,"15min":9.1210000000000004},"last":0.25700000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53900000000000003,"15min":0.72599999999999998},"min":{"1min":0.28699999999999998,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":1.3700000000000001,"5min":7.0789999999999997,"15min":8.5389999999999997},"last":2.0259999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.58199999999999996,"15min":0.747},"min":{"1min":0.34599999999999997,"5min":0.27900000000000003,"15min":0.20599999999999999},"max":{"1min":1.409,"5min":7.1130000000000004,"15min":9.1950000000000003},"last":2.0569999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739267,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11973984,"kb_used_data":10436368,"kb_used_omap":12,"kb_used_meta":1537587,"kb_avail":82397856,"statfs":{"total":96636764160,"available":84375404544,"internally_reserved":0,"allocated":10686840832,"data_stored":10686475835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1574489513},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:19:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":5.3330000000000002,"5min":5.6760000000000002,"15min":6.5780000000000003},"last":0.317},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.71699999999999997,"15min":0.65800000000000003},"min":{"1min":0.26600000000000001,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":5.4690000000000003,"5min":5.7080000000000002,"15min":6.6429999999999998},"last":0.39200000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:19:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":0.95199999999999996,"15min":0.80500000000000005},"min":{"1min":0.28399999999999997,"5min":0.26100000000000001,"15min":0.20899999999999999},"max":{"1min":5.4989999999999997,"5min":5.6280000000000001,"15min":9.3140000000000001},"last":2.7160000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.84799999999999998,"15min":0.85199999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":5.3979999999999997,"5min":5.3979999999999997,"15min":9.516},"last":3.0329999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685673472,"data_stored":10685673472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685624320,"data_stored":10685624320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685669376,"data_stored":10685669376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:20:10.203 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:10.203 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:20:30.204 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:20:30.794 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:20:30.795 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:20:30.840 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2312,"stamp":"2023-10-21T05:20:29.761599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21362444131,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":973985,"num_read_kb":3774619,"num_write":958090,"num_write_kb":3832851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64972,"ondisk_log_size":64972,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35983456,"kb_used_data":31308540,"kb_used_omap":36,"kb_used_meta":4674843,"kb_avail":247132064,"statfs":{"total":289910292480,"available":253063233536,"internally_reserved":0,"allocated":32059944960,"data_stored":32058857636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":4787039692},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,8,4,15,14,12,12,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4096,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2031,"num_read_kb":8068,"num_write":1988,"num_write_kb":7952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002087"},"pg_stats":[{"pgid":"2.7","version":"54'4893","reported_seq":7862,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:21.677998+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:20:21.677998+0000","last_peered":"2023-10-21T05:20:21.677998+0000","last_clean":"2023-10-21T05:20:21.677998+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:20:21.677998+0000","last_undegraded":"2023-10-21T05:20:21.677998+0000","last_fullsized":"2023-10-21T05:20:21.677998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4893,"log_dups_size":0,"ondisk_log_size":4893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8107","reported_seq":13207,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:21.678893+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:20:21.678893+0000","last_peered":"2023-10-21T05:20:21.678893+0000","last_clean":"2023-10-21T05:20:21.678893+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:20:21.678893+0000","last_undegraded":"2023-10-21T05:20:21.678893+0000","last_fullsized":"2023-10-21T05:20:21.678893+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8107,"log_dups_size":0,"ondisk_log_size":8107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1659","reported_seq":2729,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:20.522966+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:20:20.522966+0000","last_peered":"2023-10-21T05:20:20.522966+0000","last_clean":"2023-10-21T05:20:20.522966+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:20:20.522966+0000","last_undegraded":"2023-10-21T05:20:20.522966+0000","last_fullsized":"2023-10-21T05:20:20.522966+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1659,"log_dups_size":0,"ondisk_log_size":1659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1673","reported_seq":2864,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:20.522949+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:20:20.522949+0000","last_peered":"2023-10-21T05:20:20.522949+0000","last_clean":"2023-10-21T05:20:20.522949+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:20:20.522949+0000","last_undegraded":"2023-10-21T05:20:20.522949+0000","last_fullsized":"2023-10-21T05:20:20.522949+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1673,"log_dups_size":0,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'248546","reported_seq":499291,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:27.004566+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:20:27.004566+0000","last_peered":"2023-10-21T05:20:27.004566+0000","last_clean":"2023-10-21T05:20:27.004566+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:20:27.004566+0000","last_undegraded":"2023-10-21T05:20:27.004566+0000","last_fullsized":"2023-10-21T05:20:27.004566+0000","mapping_epoch":17,"log_start":"54'238506","ondisk_log_start":"54'238506","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523243008,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":250426,"num_read_kb":974932,"num_write":248546,"num_write_kb":994184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1718","reported_seq":2798,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:26.037464+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:20:26.037464+0000","last_peered":"2023-10-21T05:20:26.037464+0000","last_clean":"2023-10-21T05:20:26.037464+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:20:26.037464+0000","last_undegraded":"2023-10-21T05:20:26.037464+0000","last_fullsized":"2023-10-21T05:20:26.037464+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1718,"log_dups_size":0,"ondisk_log_size":1718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'232064","reported_seq":467275,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:25.530476+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:20:25.530476+0000","last_peered":"2023-10-21T05:20:25.530476+0000","last_clean":"2023-10-21T05:20:25.530476+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:20:25.530476+0000","last_undegraded":"2023-10-21T05:20:25.530476+0000","last_fullsized":"2023-10-21T05:20:25.530476+0000","mapping_epoch":17,"log_start":"54'221983","ondisk_log_start":"54'221983","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190979584,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":234907,"num_read_kb":914144,"num_write":232064,"num_write_kb":928256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1081,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:21.948074+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:20:21.948074+0000","last_peered":"2023-10-21T05:20:21.948074+0000","last_clean":"2023-10-21T05:20:21.948074+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:20:21.948074+0000","last_undegraded":"2023-10-21T05:20:21.948074+0000","last_fullsized":"2023-10-21T05:20:21.948074+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'241415","reported_seq":485479,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:26.973954+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:20:26.973954+0000","last_peered":"2023-10-21T05:20:26.973954+0000","last_clean":"2023-10-21T05:20:26.973954+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:20:26.973954+0000","last_undegraded":"2023-10-21T05:20:26.973954+0000","last_fullsized":"2023-10-21T05:20:26.973954+0000","mapping_epoch":17,"log_start":"54'231340","ondisk_log_start":"54'231340","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386178560,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":243759,"num_read_kb":949544,"num_write":241415,"num_write_kb":965660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":3999,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:21.679726+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:20:21.679726+0000","last_peered":"2023-10-21T05:20:21.679726+0000","last_clean":"2023-10-21T05:20:21.679726+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:20:21.679726+0000","last_undegraded":"2023-10-21T05:20:21.679726+0000","last_fullsized":"2023-10-21T05:20:21.679726+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'235665","reported_seq":474978,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:25.536217+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:20:25.536217+0000","last_peered":"2023-10-21T05:20:25.536217+0000","last_clean":"2023-10-21T05:20:25.536217+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:20:25.536217+0000","last_undegraded":"2023-10-21T05:20:25.536217+0000","last_fullsized":"2023-10-21T05:20:25.536217+0000","mapping_epoch":17,"log_start":"54'225573","ondisk_log_start":"54'225573","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":472,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 161s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261451283,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":239127,"num_read_kb":931092,"num_write":235665,"num_write_kb":942654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6445","reported_seq":10335,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:21.679066+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:20:21.679066+0000","last_peered":"2023-10-21T05:20:21.679066+0000","last_clean":"2023-10-21T05:20:21.679066+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:20:21.679066+0000","last_undegraded":"2023-10-21T05:20:21.679066+0000","last_fullsized":"2023-10-21T05:20:21.679066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":228,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:04.051678+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:18:04.051678+0000","last_peered":"2023-10-21T05:18:04.051678+0000","last_clean":"2023-10-21T05:18:04.051678+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:18:04.051678+0000","last_undegraded":"2023-10-21T05:18:04.051678+0000","last_fullsized":"2023-10-21T05:18:04.051678+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21361852435,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":968219,"num_read_kb":3769712,"num_write":957690,"num_write_kb":3830754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32057155584,"data_stored":32057155584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40288,"ondisk_log_size":40288,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":24585,"ondisk_log_size":24585,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739271,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11985200,"kb_used_data":10435740,"kb_used_omap":8,"kb_used_meta":1549431,"kb_avail":82386640,"statfs":{"total":96636764160,"available":84363919360,"internally_reserved":0,"allocated":10686197760,"data_stored":10685893229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1586618013},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,5,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":0.96399999999999997,"15min":0.876},"min":{"1min":0.19,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":6.0270000000000001,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.056,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.20399999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.8689999999999998,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.29099999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:19:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.58899999999999997,"15min":0.63100000000000001},"min":{"1min":0.22900000000000001,"5min":0.19700000000000001,"15min":0.186},"max":{"1min":4.6760000000000002,"5min":4.9939999999999998,"15min":19.574000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.6379999999999999,"5min":0.88400000000000001,"15min":0.94699999999999995},"min":{"1min":0.23699999999999999,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":21.234999999999999},"last":1.3640000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739269,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12006240,"kb_used_data":10436440,"kb_used_omap":15,"kb_used_meta":1569776,"kb_avail":82365600,"statfs":{"total":96636764160,"available":84342374400,"internally_reserved":0,"allocated":10686914560,"data_stored":10686496764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1607451014},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,9,7,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:32 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":0.90200000000000002,"15min":0.82599999999999996},"min":{"1min":0.191,"5min":0.16900000000000001,"15min":0.13},"max":{"1min":6.5209999999999999,"5min":13.257,"15min":14.682},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.51700000000000002,"15min":0.505},"min":{"1min":0.218,"5min":0.19600000000000001,"15min":0.18099999999999999},"max":{"1min":4.0960000000000001,"5min":4.0960000000000001,"15min":9.1210000000000004},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:19:32 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53900000000000003,"15min":0.72599999999999998},"min":{"1min":0.28699999999999998,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":1.3700000000000001,"5min":7.0789999999999997,"15min":8.5389999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.58199999999999996,"15min":0.747},"min":{"1min":0.34599999999999997,"5min":0.27900000000000003,"15min":0.20599999999999999},"max":{"1min":1.409,"5min":7.1130000000000004,"15min":9.1950000000000003},"last":0.54200000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739271,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11992016,"kb_used_data":10436360,"kb_used_omap":12,"kb_used_meta":1555635,"kb_avail":82379824,"statfs":{"total":96636764160,"available":84356939776,"internally_reserved":0,"allocated":10686832640,"data_stored":10686467643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1592970665},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,9,4,4,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:19:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.17000000000000001},"max":{"1min":5.3330000000000002,"5min":5.6760000000000002,"15min":6.5780000000000003},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.71699999999999997,"15min":0.65800000000000003},"min":{"1min":0.26600000000000001,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":5.4690000000000003,"5min":5.7080000000000002,"15min":6.6429999999999998},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Oct 21 05:19:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":0.95199999999999996,"15min":0.80500000000000005},"min":{"1min":0.28399999999999997,"5min":0.26100000000000001,"15min":0.20899999999999999},"max":{"1min":5.4989999999999997,"5min":5.6280000000000001,"15min":9.3140000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.84799999999999998,"15min":0.85199999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":5.3979999999999997,"5min":5.3979999999999997,"15min":9.516},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685710336,"data_stored":10685710336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685730816,"data_stored":10685730816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685714432,"data_stored":10685714432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:20:30.841 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:30.842 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:20:50.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:20:51.706 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:20:51.707 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:20:51.774 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2322,"stamp":"2023-10-21T05:20:49.773676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21362452323,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":977929,"num_read_kb":3790247,"num_write":962001,"num_write_kb":3848495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65070,"ondisk_log_size":65070,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":36055636,"kb_used_data":31308592,"kb_used_omap":36,"kb_used_meta":4746971,"kb_avail":247059884,"statfs":{"total":289910292480,"available":252989321216,"internally_reserved":0,"allocated":32059998208,"data_stored":32058910884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":4860898764},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,7,6,13,25,14,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1950,"num_read_kb":7708,"num_write":1969,"num_write_kb":7876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010786"},"pg_stats":[{"pgid":"2.7","version":"54'4920","reported_seq":7901,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:44.224232+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:20:44.224232+0000","last_peered":"2023-10-21T05:20:44.224232+0000","last_clean":"2023-10-21T05:20:44.224232+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:20:44.224232+0000","last_undegraded":"2023-10-21T05:20:44.224232+0000","last_fullsized":"2023-10-21T05:20:44.224232+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4920,"log_dups_size":0,"ondisk_log_size":4920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8152","reported_seq":13272,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:44.246847+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:20:44.246847+0000","last_peered":"2023-10-21T05:20:44.246847+0000","last_clean":"2023-10-21T05:20:44.246847+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:20:44.246847+0000","last_undegraded":"2023-10-21T05:20:44.246847+0000","last_fullsized":"2023-10-21T05:20:44.246847+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8152,"log_dups_size":0,"ondisk_log_size":8152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1668","reported_seq":2742,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:42.229329+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:20:42.229329+0000","last_peered":"2023-10-21T05:20:42.229329+0000","last_clean":"2023-10-21T05:20:42.229329+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:20:42.229329+0000","last_undegraded":"2023-10-21T05:20:42.229329+0000","last_fullsized":"2023-10-21T05:20:42.229329+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1668,"log_dups_size":0,"ondisk_log_size":1668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1682","reported_seq":2877,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:42.254133+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:20:42.254133+0000","last_peered":"2023-10-21T05:20:42.254133+0000","last_clean":"2023-10-21T05:20:42.254133+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:20:42.254133+0000","last_undegraded":"2023-10-21T05:20:42.254133+0000","last_fullsized":"2023-10-21T05:20:42.254133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1682,"log_dups_size":0,"ondisk_log_size":1682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'249512","reported_seq":501295,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:47.128802+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:20:47.128802+0000","last_peered":"2023-10-21T05:20:47.128802+0000","last_clean":"2023-10-21T05:20:47.128802+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:20:47.128802+0000","last_undegraded":"2023-10-21T05:20:47.128802+0000","last_fullsized":"2023-10-21T05:20:47.128802+0000","mapping_epoch":17,"log_start":"54'239416","ondisk_log_start":"54'239416","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523243008,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":251466,"num_read_kb":979048,"num_write":249512,"num_write_kb":998048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1725","reported_seq":2809,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:45.577528+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:20:45.577528+0000","last_peered":"2023-10-21T05:20:45.577528+0000","last_clean":"2023-10-21T05:20:45.577528+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:20:45.577528+0000","last_undegraded":"2023-10-21T05:20:45.577528+0000","last_fullsized":"2023-10-21T05:20:45.577528+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1725,"log_dups_size":0,"ondisk_log_size":1725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'233039","reported_seq":469213,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:45.704422+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:20:45.704422+0000","last_peered":"2023-10-21T05:20:45.704422+0000","last_clean":"2023-10-21T05:20:45.704422+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:20:45.704422+0000","last_undegraded":"2023-10-21T05:20:45.704422+0000","last_fullsized":"2023-10-21T05:20:45.704422+0000","mapping_epoch":17,"log_start":"54'222991","ondisk_log_start":"54'222991","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190979584,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":235871,"num_read_kb":917960,"num_write":233039,"num_write_kb":932156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1084,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:37.038622+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:20:37.038622+0000","last_peered":"2023-10-21T05:20:37.038622+0000","last_clean":"2023-10-21T05:20:37.038622+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:20:37.038622+0000","last_undegraded":"2023-10-21T05:20:37.038622+0000","last_fullsized":"2023-10-21T05:20:37.038622+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'242442","reported_seq":487514,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:47.119328+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:20:47.119328+0000","last_peered":"2023-10-21T05:20:47.119328+0000","last_clean":"2023-10-21T05:20:47.119328+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:20:47.119328+0000","last_undegraded":"2023-10-21T05:20:47.119328+0000","last_fullsized":"2023-10-21T05:20:47.119328+0000","mapping_epoch":17,"log_start":"54'232350","ondisk_log_start":"54'232350","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386178560,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":244767,"num_read_kb":953536,"num_write":242442,"num_write_kb":969768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4015,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:42.161577+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:20:42.161577+0000","last_peered":"2023-10-21T05:20:42.161577+0000","last_clean":"2023-10-21T05:20:42.161577+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:20:42.161577+0000","last_undegraded":"2023-10-21T05:20:42.161577+0000","last_fullsized":"2023-10-21T05:20:42.161577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'236608","reported_seq":476863,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:45.725421+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:20:45.725421+0000","last_peered":"2023-10-21T05:20:45.725421+0000","last_clean":"2023-10-21T05:20:45.725421+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:20:45.725421+0000","last_undegraded":"2023-10-21T05:20:45.725421+0000","last_fullsized":"2023-10-21T05:20:45.725421+0000","mapping_epoch":17,"log_start":"54'226591","ondisk_log_start":"54'226591","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":512,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 181s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261459475,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":240059,"num_read_kb":934796,"num_write":236608,"num_write_kb":946426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6481","reported_seq":10387,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:44.229768+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:20:44.229768+0000","last_peered":"2023-10-21T05:20:44.229768+0000","last_clean":"2023-10-21T05:20:44.229768+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:20:44.229768+0000","last_undegraded":"2023-10-21T05:20:44.229768+0000","last_fullsized":"2023-10-21T05:20:44.229768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6481,"log_dups_size":0,"ondisk_log_size":6481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":228,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:04.051678+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:18:04.051678+0000","last_peered":"2023-10-21T05:18:04.051678+0000","last_clean":"2023-10-21T05:18:04.051678+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:18:04.051678+0000","last_undegraded":"2023-10-21T05:18:04.051678+0000","last_fullsized":"2023-10-21T05:18:04.051678+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21361860627,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":972163,"num_read_kb":3785340,"num_write":961601,"num_write_kb":3846398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32057151488,"data_stored":32057151488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40253,"ondisk_log_size":40253,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":24718,"ondisk_log_size":24718,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739275,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12021292,"kb_used_data":10435744,"kb_used_omap":8,"kb_used_meta":1585527,"kb_avail":82350548,"statfs":{"total":96636764160,"available":84326961152,"internally_reserved":0,"allocated":10686201856,"data_stored":10685897325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1623580317},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":0.96399999999999997,"15min":0.876},"min":{"1min":0.19,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":6.0270000000000001,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.056,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.20399999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.8689999999999998,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.31900000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.186},"max":{"1min":4.1269999999999998,"5min":4.9939999999999998,"15min":19.574000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.81499999999999995,"15min":0.92400000000000004},"min":{"1min":0.224,"5min":0.224,"15min":0.19700000000000001},"max":{"1min":2.0600000000000001,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.36299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739273,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12024248,"kb_used_data":10436464,"kb_used_omap":15,"kb_used_meta":1587760,"kb_avail":82347592,"statfs":{"total":96636764160,"available":84323934208,"internally_reserved":0,"allocated":10686939136,"data_stored":10686521340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1625866630},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,4,4,4,12,8,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.022,"15min":0.872},"min":{"1min":0.17499999999999999,"5min":0.17299999999999999,"15min":0.13},"max":{"1min":10.311999999999999,"5min":13.257,"15min":14.682},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.54800000000000004,"15min":0.51800000000000002},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.6699999999999999,"5min":5.6699999999999999,"15min":9.1210000000000004},"last":0.23000000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:20:34 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":0.67200000000000004,"15min":0.749},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.185},"max":{"1min":14.138,"5min":14.138,"15min":14.138},"last":0.309},{"interface":"front","average":{"1min":1.2290000000000001,"5min":0.71499999999999997,"15min":0.77100000000000002},"min":{"1min":0.30399999999999999,"5min":0.29499999999999998,"15min":0.20599999999999999},"max":{"1min":14.173,"5min":14.173,"15min":14.173},"last":0.35799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739275,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12010096,"kb_used_data":10436384,"kb_used_omap":12,"kb_used_meta":1573683,"kb_avail":82361744,"statfs":{"total":96636764160,"available":84338425856,"internally_reserved":0,"allocated":10686857216,"data_stored":10686492219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1611451817},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,8,11,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:33 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.45000000000000001,"15min":0.55900000000000005},"min":{"1min":0.23100000000000001,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":0.55600000000000005,"5min":5.3330000000000002,"15min":6.5780000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.60099999999999998,"15min":0.65100000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":1.552,"5min":5.4690000000000003,"15min":6.6429999999999998},"last":0.58799999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:20:33 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.85499999999999998,"15min":0.83799999999999997},"min":{"1min":0.28899999999999998,"5min":0.26100000000000001,"15min":0.23499999999999999},"max":{"1min":6.5190000000000001,"5min":6.5190000000000001,"15min":9.3140000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.84999999999999998,"15min":0.88400000000000001},"min":{"1min":0.22,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":6.5549999999999997,"5min":6.5549999999999997,"15min":9.516},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685722624,"data_stored":10685722624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685710336,"data_stored":10685710336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685718528,"data_stored":10685718528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:20:51.776 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:51.777 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:11.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:21:12.529 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:21:12.530 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:21:12.579 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2333,"stamp":"2023-10-21T05:21:11.781784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21362472803,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":982279,"num_read_kb":3807479,"num_write":966330,"num_write_kb":3865811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65204,"ondisk_log_size":65204,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":36204788,"kb_used_data":31308572,"kb_used_omap":36,"kb_used_meta":4896155,"kb_avail":246910732,"statfs":{"total":289910292480,"available":252836589568,"internally_reserved":0,"allocated":32059977728,"data_stored":32058890404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":5013663180},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,0,8,15,20,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12288,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2357,"num_read_kb":9336,"num_write":2404,"num_write_kb":9616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006304"},"pg_stats":[{"pgid":"2.7","version":"54'4953","reported_seq":7949,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:10.185116+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:21:10.185116+0000","last_peered":"2023-10-21T05:21:10.185116+0000","last_clean":"2023-10-21T05:21:10.185116+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:21:10.185116+0000","last_undegraded":"2023-10-21T05:21:10.185116+0000","last_fullsized":"2023-10-21T05:21:10.185116+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4953,"log_dups_size":0,"ondisk_log_size":4953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8207","reported_seq":13352,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:10.206521+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:21:10.206521+0000","last_peered":"2023-10-21T05:21:10.206521+0000","last_clean":"2023-10-21T05:21:10.206521+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:21:10.206521+0000","last_undegraded":"2023-10-21T05:21:10.206521+0000","last_fullsized":"2023-10-21T05:21:10.206521+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8207,"log_dups_size":0,"ondisk_log_size":8207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1679","reported_seq":2759,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:10.174035+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:21:10.174035+0000","last_peered":"2023-10-21T05:21:10.174035+0000","last_clean":"2023-10-21T05:21:10.174035+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:21:10.174035+0000","last_undegraded":"2023-10-21T05:21:10.174035+0000","last_fullsized":"2023-10-21T05:21:10.174035+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1679,"log_dups_size":0,"ondisk_log_size":1679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1693","reported_seq":2894,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:07.753589+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:21:07.753589+0000","last_peered":"2023-10-21T05:21:07.753589+0000","last_clean":"2023-10-21T05:21:07.753589+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:21:07.753589+0000","last_undegraded":"2023-10-21T05:21:07.753589+0000","last_fullsized":"2023-10-21T05:21:07.753589+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1693,"log_dups_size":0,"ondisk_log_size":1693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'250519","reported_seq":503290,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:07.334842+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:21:07.334842+0000","last_peered":"2023-10-21T05:21:07.334842+0000","last_clean":"2023-10-21T05:21:07.334842+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:21:07.334842+0000","last_undegraded":"2023-10-21T05:21:07.334842+0000","last_fullsized":"2023-10-21T05:21:07.334842+0000","mapping_epoch":17,"log_start":"54'240428","ondisk_log_start":"54'240428","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523251200,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":252456,"num_read_kb":982956,"num_write":250519,"num_write_kb":1002076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1730","reported_seq":2818,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:02.669391+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:21:02.669391+0000","last_peered":"2023-10-21T05:21:02.669391+0000","last_clean":"2023-10-21T05:21:02.669391+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:21:02.669391+0000","last_undegraded":"2023-10-21T05:21:02.669391+0000","last_fullsized":"2023-10-21T05:21:02.669391+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1730,"log_dups_size":0,"ondisk_log_size":1730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'234222","reported_seq":471533,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:10.988579+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:21:10.988579+0000","last_peered":"2023-10-21T05:21:10.988579+0000","last_clean":"2023-10-21T05:21:10.988579+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:21:10.988579+0000","last_undegraded":"2023-10-21T05:21:10.988579+0000","last_fullsized":"2023-10-21T05:21:10.988579+0000","mapping_epoch":17,"log_start":"54'224204","ondisk_log_start":"54'224204","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190979584,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":237007,"num_read_kb":922464,"num_write":234222,"num_write_kb":936888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1088,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:57.517514+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:20:57.517514+0000","last_peered":"2023-10-21T05:20:57.517514+0000","last_clean":"2023-10-21T05:20:57.517514+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:20:57.517514+0000","last_undegraded":"2023-10-21T05:20:57.517514+0000","last_fullsized":"2023-10-21T05:20:57.517514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'243382","reported_seq":489477,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:07.366485+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:21:07.366485+0000","last_peered":"2023-10-21T05:21:07.366485+0000","last_clean":"2023-10-21T05:21:07.366485+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:21:07.366485+0000","last_undegraded":"2023-10-21T05:21:07.366485+0000","last_fullsized":"2023-10-21T05:21:07.366485+0000","mapping_epoch":17,"log_start":"54'233364","ondisk_log_start":"54'233364","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386178560,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":245792,"num_read_kb":957612,"num_write":243382,"num_write_kb":973528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4031,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:02.634314+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:21:02.634314+0000","last_peered":"2023-10-21T05:21:02.634314+0000","last_clean":"2023-10-21T05:21:02.634314+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:21:02.634314+0000","last_undegraded":"2023-10-21T05:21:02.634314+0000","last_fullsized":"2023-10-21T05:21:02.634314+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'237807","reported_seq":479276,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:10.987355+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:21:10.987355+0000","last_peered":"2023-10-21T05:21:10.987355+0000","last_clean":"2023-10-21T05:21:10.987355+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:21:10.987355+0000","last_undegraded":"2023-10-21T05:21:10.987355+0000","last_fullsized":"2023-10-21T05:21:10.987355+0000","mapping_epoch":17,"log_start":"54'227706","ondisk_log_start":"54'227706","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":587,"log_size":10101,"log_dups_size":0,"ondisk_log_size":10101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 206s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261471763,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":241258,"num_read_kb":939540,"num_write":237807,"num_write_kb":951222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6525","reported_seq":10452,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:10.250113+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:21:10.250113+0000","last_peered":"2023-10-21T05:21:10.250113+0000","last_clean":"2023-10-21T05:21:10.250113+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:21:10.250113+0000","last_undegraded":"2023-10-21T05:21:10.250113+0000","last_fullsized":"2023-10-21T05:21:10.250113+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6525,"log_dups_size":0,"ondisk_log_size":6525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":228,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:04.051678+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:18:04.051678+0000","last_peered":"2023-10-21T05:18:04.051678+0000","last_clean":"2023-10-21T05:18:04.051678+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:18:04.051678+0000","last_undegraded":"2023-10-21T05:18:04.051678+0000","last_fullsized":"2023-10-21T05:18:04.051678+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21361881107,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":976513,"num_read_kb":3802572,"num_write":965930,"num_write_kb":3863714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32057147392,"data_stored":32057147392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40228,"ondisk_log_size":40228,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":24877,"ondisk_log_size":24877,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739279,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12039984,"kb_used_data":10435752,"kb_used_omap":8,"kb_used_meta":1604215,"kb_avail":82331856,"statfs":{"total":96636764160,"available":84307820544,"internally_reserved":0,"allocated":10686210048,"data_stored":10685905517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1642716829},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:51 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":1.026,"15min":0.872},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.7779999999999996,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.439,"5min":0.74299999999999999,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.2349999999999999,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.36099999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.186},"max":{"1min":4.1269999999999998,"5min":4.9939999999999998,"15min":19.574000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.81499999999999995,"15min":0.92400000000000004},"min":{"1min":0.224,"5min":0.224,"15min":0.19700000000000001},"max":{"1min":2.0600000000000001,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.28899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739278,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12100576,"kb_used_data":10436440,"kb_used_omap":15,"kb_used_meta":1664112,"kb_avail":82271264,"statfs":{"total":96636764160,"available":84245774336,"internally_reserved":0,"allocated":10686914560,"data_stored":10686496764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1704051078},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,7,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.022,"15min":0.872},"min":{"1min":0.17499999999999999,"5min":0.17299999999999999,"15min":0.13},"max":{"1min":10.311999999999999,"5min":13.257,"15min":14.682},"last":2.161},{"interface":"front","average":{"1min":0.628,"5min":0.54800000000000004,"15min":0.51800000000000002},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.6699999999999999,"5min":5.6699999999999999,"15min":9.1210000000000004},"last":0.49099999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:20:34 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":0.67200000000000004,"15min":0.749},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.185},"max":{"1min":14.138,"5min":14.138,"15min":14.138},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.2290000000000001,"5min":0.71499999999999997,"15min":0.77100000000000002},"min":{"1min":0.30399999999999999,"5min":0.29499999999999998,"15min":0.20599999999999999},"max":{"1min":14.173,"5min":14.173,"15min":14.173},"last":0.45100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739279,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12064228,"kb_used_data":10436380,"kb_used_omap":12,"kb_used_meta":1627827,"kb_avail":82307612,"statfs":{"total":96636764160,"available":84282994688,"internally_reserved":0,"allocated":10686853120,"data_stored":10686488123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1666895273},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,2,6,5,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:33 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.45000000000000001,"15min":0.55900000000000005},"min":{"1min":0.23100000000000001,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":0.55600000000000005,"5min":5.3330000000000002,"15min":6.5780000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.60099999999999998,"15min":0.65100000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":1.552,"5min":5.4690000000000003,"15min":6.6429999999999998},"last":0.30099999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:20:33 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.85499999999999998,"15min":0.83799999999999997},"min":{"1min":0.28899999999999998,"5min":0.26100000000000001,"15min":0.23499999999999999},"max":{"1min":6.5190000000000001,"5min":6.5190000000000001,"15min":9.3140000000000001},"last":4.5620000000000003},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.84999999999999998,"15min":0.88400000000000001},"min":{"1min":0.22,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":6.5549999999999997,"5min":6.5549999999999997,"15min":9.516},"last":4.4550000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685710336,"data_stored":10685710336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685702144,"data_stored":10685702144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685734912,"data_stored":10685734912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:21:12.581 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:12.582 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:19.063 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:19.058+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:21:19.077 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:19.070+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T05:21:19.077 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:19.070+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:21:19.078 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:19.070+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T05:21:19.078 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:19.070+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:21:19.078 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:19.070+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:21:19.078 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:19.070+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:21:19.079 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:19.070+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:21:19.079 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:19.070+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T05:21:19.079 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:19.070+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T05:21:19.080 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:19.070+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:21:19.080 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:19.070+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:21:32.584 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:21:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:21:33.242 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:21:33.284 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2343,"stamp":"2023-10-21T05:21:31.789429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21362497379,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":986378,"num_read_kb":3823767,"num_write":970244,"num_write_kb":3881467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65341,"ondisk_log_size":65341,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":36295040,"kb_used_data":31308572,"kb_used_omap":36,"kb_used_meta":4986395,"kb_avail":246820480,"statfs":{"total":289910292480,"available":252744171520,"internally_reserved":0,"allocated":32059977728,"data_stored":32058890404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":5106068940},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,9,7,15,18,19,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12288,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2467,"num_read_kb":9808,"num_write":2465,"num_write_kb":9860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002030"},"pg_stats":[{"pgid":"2.7","version":"54'4965","reported_seq":7973,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:28.233553+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:21:28.233553+0000","last_peered":"2023-10-21T05:21:28.233553+0000","last_clean":"2023-10-21T05:21:28.233553+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:21:28.233553+0000","last_undegraded":"2023-10-21T05:21:28.233553+0000","last_fullsized":"2023-10-21T05:21:28.233553+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4965,"log_dups_size":0,"ondisk_log_size":4965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8227","reported_seq":13392,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:28.234074+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:21:28.234074+0000","last_peered":"2023-10-21T05:21:28.234074+0000","last_clean":"2023-10-21T05:21:28.234074+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:21:28.234074+0000","last_undegraded":"2023-10-21T05:21:28.234074+0000","last_fullsized":"2023-10-21T05:21:28.234074+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8227,"log_dups_size":0,"ondisk_log_size":8227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1683","reported_seq":2767,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:24.553926+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:21:24.553926+0000","last_peered":"2023-10-21T05:21:24.553926+0000","last_clean":"2023-10-21T05:21:24.553926+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:21:24.553926+0000","last_undegraded":"2023-10-21T05:21:24.553926+0000","last_fullsized":"2023-10-21T05:21:24.553926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1683,"log_dups_size":0,"ondisk_log_size":1683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1697","reported_seq":2902,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:24.553926+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:21:24.553926+0000","last_peered":"2023-10-21T05:21:24.553926+0000","last_clean":"2023-10-21T05:21:24.553926+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:21:24.553926+0000","last_undegraded":"2023-10-21T05:21:24.553926+0000","last_fullsized":"2023-10-21T05:21:24.553926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1697,"log_dups_size":0,"ondisk_log_size":1697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'251504","reported_seq":505343,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:27.587627+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:21:27.587627+0000","last_peered":"2023-10-21T05:21:27.587627+0000","last_clean":"2023-10-21T05:21:27.587627+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:21:27.587627+0000","last_undegraded":"2023-10-21T05:21:27.587627+0000","last_fullsized":"2023-10-21T05:21:27.587627+0000","mapping_epoch":17,"log_start":"54'241444","ondisk_log_start":"54'241444","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523251200,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":253519,"num_read_kb":987184,"num_write":251504,"num_write_kb":1006016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1739","reported_seq":2830,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:24.098084+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:21:24.098084+0000","last_peered":"2023-10-21T05:21:24.098084+0000","last_clean":"2023-10-21T05:21:24.098084+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:21:24.098084+0000","last_undegraded":"2023-10-21T05:21:24.098084+0000","last_fullsized":"2023-10-21T05:21:24.098084+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1739,"log_dups_size":0,"ondisk_log_size":1739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'235210","reported_seq":473508,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:31.190999+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:21:31.190999+0000","last_peered":"2023-10-21T05:21:31.190999+0000","last_clean":"2023-10-21T05:21:31.190999+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:21:31.190999+0000","last_undegraded":"2023-10-21T05:21:31.190999+0000","last_fullsized":"2023-10-21T05:21:31.190999+0000","mapping_epoch":17,"log_start":"54'225112","ondisk_log_start":"54'225112","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5190979584,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":237995,"num_read_kb":926408,"num_write":235210,"num_write_kb":940840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1092,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:17.994889+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:21:17.994889+0000","last_peered":"2023-10-21T05:21:17.994889+0000","last_clean":"2023-10-21T05:21:17.994889+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:21:17.994889+0000","last_undegraded":"2023-10-21T05:21:17.994889+0000","last_fullsized":"2023-10-21T05:21:17.994889+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'244354","reported_seq":491475,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:27.580174+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:21:27.580174+0000","last_peered":"2023-10-21T05:21:27.580174+0000","last_clean":"2023-10-21T05:21:27.580174+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:21:27.580174+0000","last_undegraded":"2023-10-21T05:21:27.580174+0000","last_fullsized":"2023-10-21T05:21:27.580174+0000","mapping_epoch":17,"log_start":"54'234268","ondisk_log_start":"54'234268","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386190848,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":246818,"num_read_kb":961680,"num_write":244354,"num_write_kb":977416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4047,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:23.114243+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:21:23.114243+0000","last_peered":"2023-10-21T05:21:23.114243+0000","last_clean":"2023-10-21T05:21:23.114243+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:21:23.114243+0000","last_undegraded":"2023-10-21T05:21:23.114243+0000","last_fullsized":"2023-10-21T05:21:23.114243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'238776","reported_seq":481277,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:31.237479+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:21:31.237479+0000","last_peered":"2023-10-21T05:21:31.237479+0000","last_clean":"2023-10-21T05:21:31.237479+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:21:31.237479+0000","last_undegraded":"2023-10-21T05:21:31.237479+0000","last_fullsized":"2023-10-21T05:21:31.237479+0000","mapping_epoch":17,"log_start":"54'228720","ondisk_log_start":"54'228720","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":629,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 227s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261484051,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":242280,"num_read_kb":943588,"num_write":238776,"num_write_kb":955098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6541","reported_seq":10484,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:28.233981+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:21:28.233981+0000","last_peered":"2023-10-21T05:21:28.233981+0000","last_clean":"2023-10-21T05:21:28.233981+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:21:28.233981+0000","last_undegraded":"2023-10-21T05:21:28.233981+0000","last_fullsized":"2023-10-21T05:21:28.233981+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6541,"log_dups_size":0,"ondisk_log_size":6541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":228,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:04.051678+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:18:04.051678+0000","last_peered":"2023-10-21T05:18:04.051678+0000","last_clean":"2023-10-21T05:18:04.051678+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:18:04.051678+0000","last_undegraded":"2023-10-21T05:18:04.051678+0000","last_fullsized":"2023-10-21T05:18:04.051678+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21361905683,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":980612,"num_read_kb":3818860,"num_write":969844,"num_write_kb":3879370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32057147392,"data_stored":32057147392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40300,"ondisk_log_size":40300,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":24942,"ondisk_log_size":24942,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739283,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12094136,"kb_used_data":10435760,"kb_used_omap":8,"kb_used_meta":1658359,"kb_avail":82277704,"statfs":{"total":96636764160,"available":84252368896,"internally_reserved":0,"allocated":10686218240,"data_stored":10685913709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1698160285},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:51 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":1.026,"15min":0.872},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.7779999999999996,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.74299999999999999,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.2349999999999999,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.22700000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.186},"max":{"1min":4.1269999999999998,"5min":4.9939999999999998,"15min":19.574000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.81499999999999995,"15min":0.92400000000000004},"min":{"1min":0.224,"5min":0.224,"15min":0.19700000000000001},"max":{"1min":2.0600000000000001,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.34899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739282,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12118636,"kb_used_data":10436440,"kb_used_omap":15,"kb_used_meta":1682160,"kb_avail":82253204,"statfs":{"total":96636764160,"available":84227280896,"internally_reserved":0,"allocated":10686914560,"data_stored":10686496764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1722532230},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,6,12,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.022,"15min":0.872},"min":{"1min":0.17499999999999999,"5min":0.17299999999999999,"15min":0.13},"max":{"1min":10.311999999999999,"5min":13.257,"15min":14.682},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.54800000000000004,"15min":0.51800000000000002},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.6699999999999999,"5min":5.6699999999999999,"15min":9.1210000000000004},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:20:34 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":0.67200000000000004,"15min":0.749},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.185},"max":{"1min":14.138,"5min":14.138,"15min":14.138},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.2290000000000001,"5min":0.71499999999999997,"15min":0.77100000000000002},"min":{"1min":0.30399999999999999,"5min":0.29499999999999998,"15min":0.20599999999999999},"max":{"1min":14.173,"5min":14.173,"15min":14.173},"last":0.373}]}]},{"osd":0,"up_from":8,"seq":34359739283,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12082268,"kb_used_data":10436372,"kb_used_omap":12,"kb_used_meta":1645875,"kb_avail":82289572,"statfs":{"total":96636764160,"available":84264521728,"internally_reserved":0,"allocated":10686844928,"data_stored":10686479931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1685376425},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,6,5,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:33 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.45000000000000001,"15min":0.55900000000000005},"min":{"1min":0.23100000000000001,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":0.55600000000000005,"5min":5.3330000000000002,"15min":6.5780000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.60099999999999998,"15min":0.65100000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":1.552,"5min":5.4690000000000003,"15min":6.6429999999999998},"last":0.498}]},{"osd":2,"last update":"Sat Oct 21 05:20:33 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.85499999999999998,"15min":0.83799999999999997},"min":{"1min":0.28899999999999998,"5min":0.26100000000000001,"15min":0.23499999999999999},"max":{"1min":6.5190000000000001,"5min":6.5190000000000001,"15min":9.3140000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.84999999999999998,"15min":0.88400000000000001},"min":{"1min":0.22,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":6.5549999999999997,"5min":6.5549999999999997,"15min":9.516},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685706240,"data_stored":10685706240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685730816,"data_stored":10685730816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685710336,"data_stored":10685710336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:21:33.286 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:33.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:21:34.143 INFO:teuthology.orchestra.run.smithi172.stderr:instructing pg 3.2s0 on osd.1 to deep-scrub 2023-10-21T05:21:34.228 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:54.229 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:21:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:21:54.924 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:21:54.965 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2354,"stamp":"2023-10-21T05:21:53.794996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21362677603,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":990856,"num_read_kb":3841591,"num_write":974659,"num_write_kb":3899127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65426,"ondisk_log_size":65426,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":36375692,"kb_used_data":31308788,"kb_used_omap":36,"kb_used_meta":5066843,"kb_avail":246739828,"statfs":{"total":289910292480,"available":252661583872,"internally_reserved":0,"allocated":32060198912,"data_stored":32059111588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":5188447692},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,7,15,13,16,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":155648,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2491,"num_read_kb":9920,"num_write":2469,"num_write_kb":9876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001998"},"pg_stats":[{"pgid":"2.7","version":"54'4992","reported_seq":8013,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:50.765830+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:21:50.765830+0000","last_peered":"2023-10-21T05:21:50.765830+0000","last_clean":"2023-10-21T05:21:50.765830+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:21:50.765830+0000","last_undegraded":"2023-10-21T05:21:50.765830+0000","last_fullsized":"2023-10-21T05:21:50.765830+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":4992,"log_dups_size":0,"ondisk_log_size":4992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8272","reported_seq":13458,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:50.784611+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:21:50.784611+0000","last_peered":"2023-10-21T05:21:50.784611+0000","last_clean":"2023-10-21T05:21:50.784611+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:21:50.784611+0000","last_undegraded":"2023-10-21T05:21:50.784611+0000","last_fullsized":"2023-10-21T05:21:50.784611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8272,"log_dups_size":0,"ondisk_log_size":8272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1692","reported_seq":2780,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:49.498366+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:21:49.498366+0000","last_peered":"2023-10-21T05:21:49.498366+0000","last_clean":"2023-10-21T05:21:49.498366+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:21:49.498366+0000","last_undegraded":"2023-10-21T05:21:49.498366+0000","last_fullsized":"2023-10-21T05:21:49.498366+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1692,"log_dups_size":0,"ondisk_log_size":1692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1706","reported_seq":2915,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:49.513072+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:21:49.513072+0000","last_peered":"2023-10-21T05:21:49.513072+0000","last_clean":"2023-10-21T05:21:49.513072+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:21:49.513072+0000","last_undegraded":"2023-10-21T05:21:49.513072+0000","last_fullsized":"2023-10-21T05:21:49.513072+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1706,"log_dups_size":0,"ondisk_log_size":1706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'252812","reported_seq":507970,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:52.877336+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:21:52.877336+0000","last_peered":"2023-10-21T05:21:52.877336+0000","last_clean":"2023-10-21T05:21:52.877336+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:21:52.877336+0000","last_undegraded":"2023-10-21T05:21:52.877336+0000","last_fullsized":"2023-10-21T05:21:52.877336+0000","mapping_epoch":17,"log_start":"54'242770","ondisk_log_start":"54'242770","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523296256,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":254838,"num_read_kb":992432,"num_write":252812,"num_write_kb":1011248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1755","reported_seq":2851,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:50.402785+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:21:50.402785+0000","last_peered":"2023-10-21T05:21:50.402785+0000","last_clean":"2023-10-21T05:21:50.402785+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:21:50.402785+0000","last_undegraded":"2023-10-21T05:21:50.402785+0000","last_fullsized":"2023-10-21T05:21:50.402785+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1755,"log_dups_size":0,"ondisk_log_size":1755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'236171","reported_seq":475446,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:51.451730+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:21:51.451730+0000","last_peered":"2023-10-21T05:21:51.451730+0000","last_clean":"2023-10-21T05:21:51.451730+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:21:51.451730+0000","last_undegraded":"2023-10-21T05:21:51.451730+0000","last_fullsized":"2023-10-21T05:21:51.451730+0000","mapping_epoch":17,"log_start":"54'226126","ondisk_log_start":"54'226126","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5191028736,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":238975,"num_read_kb":930308,"num_write":236171,"num_write_kb":944684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1097,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:43.598431+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:21:43.598431+0000","last_peered":"2023-10-21T05:21:43.598431+0000","last_clean":"2023-10-21T05:21:43.598431+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:21:43.598431+0000","last_undegraded":"2023-10-21T05:21:43.598431+0000","last_fullsized":"2023-10-21T05:21:43.598431+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'245536","reported_seq":493901,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:52.835282+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:21:52.835282+0000","last_peered":"2023-10-21T05:21:52.835282+0000","last_clean":"2023-10-21T05:21:52.835282+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:21:52.835282+0000","last_undegraded":"2023-10-21T05:21:52.835282+0000","last_fullsized":"2023-10-21T05:21:52.835282+0000","mapping_epoch":17,"log_start":"54'235479","ondisk_log_start":"54'235479","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386219520,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":248059,"num_read_kb":966612,"num_write":245536,"num_write_kb":982144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4068,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:48.716167+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:21:48.716167+0000","last_peered":"2023-10-21T05:21:48.716167+0000","last_clean":"2023-10-21T05:21:48.716167+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:21:48.716167+0000","last_undegraded":"2023-10-21T05:21:48.716167+0000","last_fullsized":"2023-10-21T05:21:48.716167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'239740","reported_seq":483188,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:51.445232+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:21:51.445232+0000","last_peered":"2023-10-21T05:21:51.445232+0000","last_clean":"2023-10-21T05:21:51.445232+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:21:51.445232+0000","last_undegraded":"2023-10-21T05:21:51.445232+0000","last_fullsized":"2023-10-21T05:21:51.445232+0000","mapping_epoch":17,"log_start":"54'229641","ondisk_log_start":"54'229641","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":689,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 247s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261541395,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":243218,"num_read_kb":947332,"num_write":239740,"num_write_kb":958954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6577","reported_seq":10536,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:21:50.811977+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:21:50.811977+0000","last_peered":"2023-10-21T05:21:50.811977+0000","last_clean":"2023-10-21T05:21:50.811977+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:21:50.811977+0000","last_undegraded":"2023-10-21T05:21:50.811977+0000","last_fullsized":"2023-10-21T05:21:50.811977+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6577,"log_dups_size":0,"ondisk_log_size":6577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":228,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:18:04.051678+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:18:04.051678+0000","last_peered":"2023-10-21T05:18:04.051678+0000","last_clean":"2023-10-21T05:18:04.051678+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:18:04.051678+0000","last_undegraded":"2023-10-21T05:18:04.051678+0000","last_fullsized":"2023-10-21T05:18:04.051678+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21362085907,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":985090,"num_read_kb":3836684,"num_write":974259,"num_write_kb":3897030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32057389056,"data_stored":32057389056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40243,"ondisk_log_size":40243,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":25084,"ondisk_log_size":25084,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739288,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12116416,"kb_used_data":10435832,"kb_used_omap":8,"kb_used_meta":1680567,"kb_avail":82255424,"statfs":{"total":96636764160,"available":84229554176,"internally_reserved":0,"allocated":10686291968,"data_stored":10685987437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1720901277},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:51 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":1.026,"15min":0.872},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.7779999999999996,"5min":11.904999999999999,"15min":22.969000000000001},"last":4.4009999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.74299999999999999,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.2349999999999999,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.28100000000000003}]},{"osd":1,"last update":"Sat Oct 21 05:21:38 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.71999999999999997,"15min":0.66100000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":19.574000000000002},"last":4.4530000000000003},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.89600000000000002,"15min":0.94999999999999996},"min":{"1min":0.26300000000000001,"5min":0.224,"15min":0.19700000000000001},"max":{"1min":6.8440000000000003,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.248}]}]},{"osd":1,"up_from":8,"seq":34359739286,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12136648,"kb_used_data":10436480,"kb_used_omap":15,"kb_used_meta":1700144,"kb_avail":82235192,"statfs":{"total":96636764160,"available":84208836608,"internally_reserved":0,"allocated":10686955520,"data_stored":10686537724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1740947846},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,6,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":1.0029999999999999,"15min":0.86099999999999999},"min":{"1min":0.19700000000000001,"5min":0.17299999999999999,"15min":0.13},"max":{"1min":3.5800000000000001,"5min":13.257,"15min":14.682},"last":0.249},{"interface":"front","average":{"1min":0.623,"5min":0.59799999999999998,"15min":0.52500000000000002},"min":{"1min":0.19,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.9710000000000001,"5min":5.6699999999999999,"15min":9.1210000000000004},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.71899999999999997,"15min":0.72499999999999998},"min":{"1min":0.26700000000000002,"5min":0.23599999999999999,"15min":0.185},"max":{"1min":4.2859999999999996,"5min":14.138,"15min":14.138},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.76000000000000001,"15min":0.73599999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.20599999999999999},"max":{"1min":4.3280000000000003,"5min":14.173,"15min":14.173},"last":0.40100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739288,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12122628,"kb_used_data":10436476,"kb_used_omap":12,"kb_used_meta":1686131,"kb_avail":82249212,"statfs":{"total":96636764160,"available":84223193088,"internally_reserved":0,"allocated":10686951424,"data_stored":10686586427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1726598569},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,11,5,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.432,"15min":0.55100000000000005},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":0.61599999999999999,"5min":5.3330000000000002,"15min":6.5780000000000003},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.626,"15min":0.66000000000000003},"min":{"1min":0.23000000000000001,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":3.7749999999999999,"5min":5.4690000000000003,"15min":6.6429999999999998},"last":0.36799999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.88400000000000001,"15min":0.872},"min":{"1min":0.28000000000000003,"5min":0.26200000000000001,"15min":0.23499999999999999},"max":{"1min":4.5620000000000003,"5min":6.5190000000000001,"15min":9.3140000000000001},"last":3.0150000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.84699999999999998,"15min":0.89000000000000001},"min":{"1min":0.25,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":4.4550000000000001,"5min":6.5549999999999997,"15min":9.516},"last":3.1190000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685792256,"data_stored":10685792256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685816832,"data_stored":10685816832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685779968,"data_stored":10685779968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:21:54.967 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:54.968 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:57.707 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:57.697+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T05:21:57.707 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:57.697+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:21:57.707 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:57.697+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:21:57.708 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:57.697+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:21:57.708 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:57.697+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:21:57.708 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:57.697+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:21:57.709 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:57.701+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:21:57.709 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:57.701+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:21:57.710 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:21:57.701+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:22:14.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:22:15.670 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:22:15.672 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:22:15.724 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2364,"stamp":"2023-10-21T05:22:13.798453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21362956131,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":994802,"num_read_kb":3857187,"num_write":978600,"num_write_kb":3914891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65488,"ondisk_log_size":65488,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":36484380,"kb_used_data":31309188,"kb_used_omap":36,"kb_used_meta":5175131,"kb_avail":246631140,"statfs":{"total":289910292480,"available":252550287360,"internally_reserved":0,"allocated":32060608512,"data_stored":32059521188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":5299334604},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,18,17,15,19,26,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":81920,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2522,"num_read_kb":9956,"num_write":2472,"num_write_kb":9888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001995"},"pg_stats":[{"pgid":"2.7","version":"54'5007","reported_seq":8040,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:10.709382+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:22:10.709382+0000","last_peered":"2023-10-21T05:22:10.709382+0000","last_clean":"2023-10-21T05:22:10.709382+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:22:10.709382+0000","last_undegraded":"2023-10-21T05:22:10.709382+0000","last_fullsized":"2023-10-21T05:22:10.709382+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8297","reported_seq":13503,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:10.716753+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:22:10.716753+0000","last_peered":"2023-10-21T05:22:10.716753+0000","last_clean":"2023-10-21T05:22:10.716753+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:22:10.716753+0000","last_undegraded":"2023-10-21T05:22:10.716753+0000","last_fullsized":"2023-10-21T05:22:10.716753+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8297,"log_dups_size":0,"ondisk_log_size":8297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1697","reported_seq":2789,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:10.405082+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:22:10.405082+0000","last_peered":"2023-10-21T05:22:10.405082+0000","last_clean":"2023-10-21T05:22:10.405082+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:22:10.405082+0000","last_undegraded":"2023-10-21T05:22:10.405082+0000","last_fullsized":"2023-10-21T05:22:10.405082+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1697,"log_dups_size":0,"ondisk_log_size":1697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1711","reported_seq":2924,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:10.408851+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:22:10.408851+0000","last_peered":"2023-10-21T05:22:10.408851+0000","last_clean":"2023-10-21T05:22:10.408851+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:22:10.408851+0000","last_undegraded":"2023-10-21T05:22:10.408851+0000","last_fullsized":"2023-10-21T05:22:10.408851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1711,"log_dups_size":0,"ondisk_log_size":1711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'253815","reported_seq":509976,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:13.046748+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:22:13.046748+0000","last_peered":"2023-10-21T05:22:13.046748+0000","last_clean":"2023-10-21T05:22:13.046748+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:22:13.046748+0000","last_undegraded":"2023-10-21T05:22:13.046748+0000","last_fullsized":"2023-10-21T05:22:13.046748+0000","mapping_epoch":17,"log_start":"54'243782","ondisk_log_start":"54'243782","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523312640,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":255843,"num_read_kb":996404,"num_write":253815,"num_write_kb":1015260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1759","reported_seq":2859,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:10.364512+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:22:10.364512+0000","last_peered":"2023-10-21T05:22:10.364512+0000","last_clean":"2023-10-21T05:22:10.364512+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:22:10.364512+0000","last_undegraded":"2023-10-21T05:22:10.364512+0000","last_fullsized":"2023-10-21T05:22:10.364512+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1759,"log_dups_size":0,"ondisk_log_size":1759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'237093","reported_seq":477370,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:11.637407+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:22:11.637407+0000","last_peered":"2023-10-21T05:22:11.637407+0000","last_clean":"2023-10-21T05:22:11.637407+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:22:11.637407+0000","last_undegraded":"2023-10-21T05:22:11.637407+0000","last_fullsized":"2023-10-21T05:22:11.637407+0000","mapping_epoch":17,"log_start":"54'227041","ondisk_log_start":"54'227041","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5191110656,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":239971,"num_read_kb":934236,"num_write":237093,"num_write_kb":948372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1101,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:04.090115+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:22:04.090115+0000","last_peered":"2023-10-21T05:22:04.090115+0000","last_clean":"2023-10-21T05:22:04.090115+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:22:04.090115+0000","last_undegraded":"2023-10-21T05:22:04.090115+0000","last_fullsized":"2023-10-21T05:22:04.090115+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'246556","reported_seq":495930,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:13.025572+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:22:13.025572+0000","last_peered":"2023-10-21T05:22:13.025572+0000","last_clean":"2023-10-21T05:22:13.025572+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:22:13.025572+0000","last_undegraded":"2023-10-21T05:22:13.025572+0000","last_fullsized":"2023-10-21T05:22:13.025572+0000","mapping_epoch":17,"log_start":"54'236495","ondisk_log_start":"54'236495","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386309632,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":249068,"num_read_kb":970584,"num_write":246556,"num_write_kb":986224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4084,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:09.200062+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:22:09.200062+0000","last_peered":"2023-10-21T05:22:09.200062+0000","last_clean":"2023-10-21T05:22:09.200062+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:22:09.200062+0000","last_undegraded":"2023-10-21T05:22:09.200062+0000","last_fullsized":"2023-10-21T05:22:09.200062+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'240736","reported_seq":485132,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:11.646620+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:22:11.646620+0000","last_peered":"2023-10-21T05:22:11.646620+0000","last_clean":"2023-10-21T05:22:11.646620+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:22:11.646620+0000","last_undegraded":"2023-10-21T05:22:11.646620+0000","last_fullsized":"2023-10-21T05:22:11.646620+0000","mapping_epoch":17,"log_start":"54'230650","ondisk_log_start":"54'230650","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":730,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 267s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261631507,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":244154,"num_read_kb":951056,"num_write":240736,"num_write_kb":962938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6596","reported_seq":10571,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:10.778501+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:22:10.778501+0000","last_peered":"2023-10-21T05:22:10.778501+0000","last_clean":"2023-10-21T05:22:10.778501+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:22:10.778501+0000","last_undegraded":"2023-10-21T05:22:10.778501+0000","last_fullsized":"2023-10-21T05:22:10.778501+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":229,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:05.382421+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:20:05.382421+0000","last_peered":"2023-10-21T05:20:05.382421+0000","last_clean":"2023-10-21T05:20:05.382421+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:20:05.382421+0000","last_undegraded":"2023-10-21T05:20:05.382421+0000","last_fullsized":"2023-10-21T05:20:05.382421+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21362364435,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":989036,"num_read_kb":3852280,"num_write":978200,"num_write_kb":3912794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32057696256,"data_stored":32057696256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40232,"ondisk_log_size":40232,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":25157,"ondisk_log_size":25157,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739292,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12152616,"kb_used_data":10435936,"kb_used_omap":8,"kb_used_meta":1716663,"kb_avail":82219224,"statfs":{"total":96636764160,"available":84192485376,"internally_reserved":0,"allocated":10686398464,"data_stored":10686093933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1757863581},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,14,7,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.0920000000000001,"15min":0.89800000000000002},"min":{"1min":0.20699999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.7229999999999999,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.73399999999999999,"15min":0.73299999999999998},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.17899999999999999},"max":{"1min":4.3780000000000001,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.39900000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:21:38 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.71999999999999997,"15min":0.66100000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":19.574000000000002},"last":0.31},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.89600000000000002,"15min":0.94999999999999996},"min":{"1min":0.26300000000000001,"5min":0.224,"15min":0.19700000000000001},"max":{"1min":6.8440000000000003,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.372}]}]},{"osd":1,"up_from":8,"seq":34359739290,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12172924,"kb_used_data":10436660,"kb_used_omap":15,"kb_used_meta":1736240,"kb_avail":82198916,"statfs":{"total":96636764160,"available":84171689984,"internally_reserved":0,"allocated":10687139840,"data_stored":10686722044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1777910150},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,8,4,12,13,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":1.0029999999999999,"15min":0.86099999999999999},"min":{"1min":0.19700000000000001,"5min":0.17299999999999999,"15min":0.13},"max":{"1min":3.5800000000000001,"5min":13.257,"15min":14.682},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.59799999999999998,"15min":0.52500000000000002},"min":{"1min":0.19,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.9710000000000001,"5min":5.6699999999999999,"15min":9.1210000000000004},"last":0.309}]},{"osd":2,"last update":"Sat Oct 21 05:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.71899999999999997,"15min":0.72499999999999998},"min":{"1min":0.26700000000000002,"5min":0.23599999999999999,"15min":0.185},"max":{"1min":4.2859999999999996,"5min":14.138,"15min":14.138},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.76000000000000001,"15min":0.73599999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.20599999999999999},"max":{"1min":4.3280000000000003,"5min":14.173,"15min":14.173},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739292,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12158840,"kb_used_data":10436592,"kb_used_omap":12,"kb_used_meta":1722227,"kb_avail":82213000,"statfs":{"total":96636764160,"available":84186112000,"internally_reserved":0,"allocated":10687070208,"data_stored":10686705211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1763560873},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2,3,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.432,"15min":0.55100000000000005},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":0.61599999999999999,"5min":5.3330000000000002,"15min":6.5780000000000003},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.626,"15min":0.66000000000000003},"min":{"1min":0.23000000000000001,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":3.7749999999999999,"5min":5.4690000000000003,"15min":6.6429999999999998},"last":0.48499999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.88400000000000001,"15min":0.872},"min":{"1min":0.28000000000000003,"5min":0.26200000000000001,"15min":0.23499999999999999},"max":{"1min":4.5620000000000003,"5min":6.5190000000000001,"15min":9.3140000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.84699999999999998,"15min":0.89000000000000001},"min":{"1min":0.25,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":4.4550000000000001,"5min":6.5549999999999997,"15min":9.516},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10685890560,"data_stored":10685890560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10685878272,"data_stored":10685878272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10685927424,"data_stored":10685927424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:22:15.726 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:22:15.726 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:22:35.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:22:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:22:36.387 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:22:36.419 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2375,"stamp":"2023-10-21T05:22:35.807112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21363238755,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":998594,"num_read_kb":3872155,"num_write":982543,"num_write_kb":3930663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65529,"ondisk_log_size":65529,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":36593000,"kb_used_data":31309520,"kb_used_omap":36,"kb_used_meta":5283419,"kb_avail":246522520,"statfs":{"total":289910292480,"available":252439060480,"internally_reserved":0,"allocated":32060948480,"data_stored":32059861156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":5410221516},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,9,13,10,24,6,17,17],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":81920,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1848,"num_read_kb":7272,"num_write":1961,"num_write_kb":7844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002121"},"pg_stats":[{"pgid":"2.7","version":"54'5022","reported_seq":8067,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:29.673613+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:22:29.673613+0000","last_peered":"2023-10-21T05:22:29.673613+0000","last_clean":"2023-10-21T05:22:29.673613+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:22:29.673613+0000","last_undegraded":"2023-10-21T05:22:29.673613+0000","last_fullsized":"2023-10-21T05:22:29.673613+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":5022,"log_dups_size":0,"ondisk_log_size":5022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8322","reported_seq":13548,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:29.674411+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:22:29.674411+0000","last_peered":"2023-10-21T05:22:29.674411+0000","last_clean":"2023-10-21T05:22:29.674411+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:22:29.674411+0000","last_undegraded":"2023-10-21T05:22:29.674411+0000","last_fullsized":"2023-10-21T05:22:29.674411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8322,"log_dups_size":0,"ondisk_log_size":8322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1702","reported_seq":2798,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:28.635285+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:22:28.635285+0000","last_peered":"2023-10-21T05:22:28.635285+0000","last_clean":"2023-10-21T05:22:28.635285+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:22:28.635285+0000","last_undegraded":"2023-10-21T05:22:28.635285+0000","last_fullsized":"2023-10-21T05:22:28.635285+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1702,"log_dups_size":0,"ondisk_log_size":1702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1716","reported_seq":2933,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:28.636338+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:22:28.636338+0000","last_peered":"2023-10-21T05:22:28.636338+0000","last_clean":"2023-10-21T05:22:28.636338+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:22:28.636338+0000","last_undegraded":"2023-10-21T05:22:28.636338+0000","last_fullsized":"2023-10-21T05:22:28.636338+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1716,"log_dups_size":0,"ondisk_log_size":1716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'254876","reported_seq":512052,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:33.335226+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:22:33.335226+0000","last_peered":"2023-10-21T05:22:33.335226+0000","last_clean":"2023-10-21T05:22:33.335226+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:22:33.335226+0000","last_undegraded":"2023-10-21T05:22:33.335226+0000","last_fullsized":"2023-10-21T05:22:33.335226+0000","mapping_epoch":17,"log_start":"54'244806","ondisk_log_start":"54'244806","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523398656,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":256860,"num_read_kb":1000436,"num_write":254876,"num_write_kb":1019504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1764","reported_seq":2868,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:31.749173+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:22:31.749173+0000","last_peered":"2023-10-21T05:22:31.749173+0000","last_clean":"2023-10-21T05:22:31.749173+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:22:31.749173+0000","last_undegraded":"2023-10-21T05:22:31.749173+0000","last_fullsized":"2023-10-21T05:22:31.749173+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1764,"log_dups_size":0,"ondisk_log_size":1764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'238076","reported_seq":479211,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:31.931377+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:22:31.931377+0000","last_peered":"2023-10-21T05:22:31.931377+0000","last_clean":"2023-10-21T05:22:31.931377+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:22:31.931377+0000","last_undegraded":"2023-10-21T05:22:31.931377+0000","last_fullsized":"2023-10-21T05:22:31.931377+0000","mapping_epoch":17,"log_start":"54'228070","ondisk_log_start":"54'228070","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5191168000,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":240833,"num_read_kb":937632,"num_write":238076,"num_write_kb":952304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1106,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:27.916165+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:22:27.916165+0000","last_peered":"2023-10-21T05:22:27.916165+0000","last_clean":"2023-10-21T05:22:27.916165+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:22:27.916165+0000","last_undegraded":"2023-10-21T05:22:27.916165+0000","last_fullsized":"2023-10-21T05:22:27.916165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'247540","reported_seq":497846,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:33.335744+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:22:33.335744+0000","last_peered":"2023-10-21T05:22:33.335744+0000","last_clean":"2023-10-21T05:22:33.335744+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:22:33.335744+0000","last_undegraded":"2023-10-21T05:22:33.335744+0000","last_fullsized":"2023-10-21T05:22:33.335744+0000","mapping_epoch":17,"log_start":"54'237501","ondisk_log_start":"54'237501","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386399744,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":250000,"num_read_kb":974224,"num_write":247540,"num_write_kb":990160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4100,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:29.680285+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:22:29.680285+0000","last_peered":"2023-10-21T05:22:29.680285+0000","last_clean":"2023-10-21T05:22:29.680285+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:22:29.680285+0000","last_undegraded":"2023-10-21T05:22:29.680285+0000","last_fullsized":"2023-10-21T05:22:29.680285+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'241651","reported_seq":487036,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:31.902307+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:22:31.902307+0000","last_peered":"2023-10-21T05:22:31.902307+0000","last_clean":"2023-10-21T05:22:31.902307+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:22:31.902307+0000","last_undegraded":"2023-10-21T05:22:31.902307+0000","last_fullsized":"2023-10-21T05:22:31.902307+0000","mapping_epoch":17,"log_start":"54'231568","ondisk_log_start":"54'231568","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":778,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 287s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261680659,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":245135,"num_read_kb":954956,"num_write":241651,"num_write_kb":966598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6616","reported_seq":10607,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:29.675377+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:22:29.675377+0000","last_peered":"2023-10-21T05:22:29.675377+0000","last_clean":"2023-10-21T05:22:29.675377+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:22:29.675377+0000","last_undegraded":"2023-10-21T05:22:29.675377+0000","last_fullsized":"2023-10-21T05:22:29.675377+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6616,"log_dups_size":0,"ondisk_log_size":6616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":229,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:05.382421+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:20:05.382421+0000","last_peered":"2023-10-21T05:20:05.382421+0000","last_clean":"2023-10-21T05:20:05.382421+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:20:05.382421+0000","last_undegraded":"2023-10-21T05:20:05.382421+0000","last_fullsized":"2023-10-21T05:20:05.382421+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21362647059,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":992828,"num_read_kb":3867248,"num_write":982143,"num_write_kb":3928566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32058097664,"data_stored":32058097664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40198,"ondisk_log_size":40198,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":25232,"ondisk_log_size":25232,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12188852,"kb_used_data":10436076,"kb_used_omap":8,"kb_used_meta":1752759,"kb_avail":82182988,"statfs":{"total":96636764160,"available":84155379712,"internally_reserved":0,"allocated":10686541824,"data_stored":10686237293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1794825885},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.0920000000000001,"15min":0.89800000000000002},"min":{"1min":0.20699999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.7229999999999999,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.184},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.73399999999999999,"15min":0.73299999999999998},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.17899999999999999},"max":{"1min":4.3780000000000001,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.25600000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:21:38 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.71999999999999997,"15min":0.66100000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":19.574000000000002},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.89600000000000002,"15min":0.94999999999999996},"min":{"1min":0.26300000000000001,"5min":0.224,"15min":0.19700000000000001},"max":{"1min":6.8440000000000003,"5min":8.3840000000000003,"15min":21.234999999999999},"last":2.2690000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739294,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12209108,"kb_used_data":10436748,"kb_used_omap":15,"kb_used_meta":1772336,"kb_avail":82162732,"statfs":{"total":96636764160,"available":84134637568,"internally_reserved":0,"allocated":10687229952,"data_stored":10686812156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1814872454},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,1,17,2,7,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":1.0029999999999999,"15min":0.86099999999999999},"min":{"1min":0.19700000000000001,"5min":0.17299999999999999,"15min":0.13},"max":{"1min":3.5800000000000001,"5min":13.257,"15min":14.682},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.59799999999999998,"15min":0.52500000000000002},"min":{"1min":0.19,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.9710000000000001,"5min":5.6699999999999999,"15min":9.1210000000000004},"last":0.29599999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.71899999999999997,"15min":0.72499999999999998},"min":{"1min":0.26700000000000002,"5min":0.23599999999999999,"15min":0.185},"max":{"1min":4.2859999999999996,"5min":14.138,"15min":14.138},"last":4.8719999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.76000000000000001,"15min":0.73599999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.20599999999999999},"max":{"1min":4.3280000000000003,"5min":14.173,"15min":14.173},"last":4.907}]}]},{"osd":0,"up_from":8,"seq":34359739296,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12195040,"kb_used_data":10436696,"kb_used_omap":12,"kb_used_meta":1758323,"kb_avail":82176800,"statfs":{"total":96636764160,"available":84149043200,"internally_reserved":0,"allocated":10687176704,"data_stored":10686811707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1800523177},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,9,5,6,3,10,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.432,"15min":0.55100000000000005},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":0.61599999999999999,"5min":5.3330000000000002,"15min":6.5780000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.626,"15min":0.66000000000000003},"min":{"1min":0.23000000000000001,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":3.7749999999999999,"5min":5.4690000000000003,"15min":6.6429999999999998},"last":0.52600000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.88400000000000001,"15min":0.872},"min":{"1min":0.28000000000000003,"5min":0.26200000000000001,"15min":0.23499999999999999},"max":{"1min":4.5620000000000003,"5min":6.5190000000000001,"15min":9.3140000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.84699999999999998,"15min":0.89000000000000001},"min":{"1min":0.25,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":4.4550000000000001,"5min":6.5549999999999997,"15min":9.516},"last":0.54900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10686046208,"data_stored":10686046208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10686017536,"data_stored":10686017536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10686033920,"data_stored":10686033920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:22:36.421 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:22:36.421 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:22:39.592 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:22:39.584+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:22:39.598 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:22:39.592+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:22:39.599 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:22:39.596+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:22:39.599 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:22:39.596+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:22:39.604 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:22:39.600+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:22:39.604 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:22:39.600+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:22:39.604 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:22:39.600+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:22:56.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:22:57.232 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:22:57.233 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:22:57.291 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2385,"stamp":"2023-10-21T05:22:55.817002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21363529571,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":1002341,"num_read_kb":3886943,"num_write":986396,"num_write_kb":3946075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65596,"ondisk_log_size":65596,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":36647660,"kb_used_data":31310016,"kb_used_omap":36,"kb_used_meta":5337563,"kb_avail":246467860,"statfs":{"total":289910292480,"available":252383088640,"internally_reserved":0,"allocated":32061456384,"data_stored":32060369060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":5465664972},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,8,13,12,7,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":184320,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1797,"num_read_kb":7112,"num_write":1958,"num_write_kb":7832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008236"},"pg_stats":[{"pgid":"2.7","version":"54'5037","reported_seq":8094,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:50.153702+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:22:50.153702+0000","last_peered":"2023-10-21T05:22:50.153702+0000","last_clean":"2023-10-21T05:22:50.153702+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:22:50.153702+0000","last_undegraded":"2023-10-21T05:22:50.153702+0000","last_fullsized":"2023-10-21T05:22:50.153702+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":5037,"log_dups_size":0,"ondisk_log_size":5037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8345","reported_seq":13591,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:50.154687+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:22:50.154687+0000","last_peered":"2023-10-21T05:22:50.154687+0000","last_clean":"2023-10-21T05:22:50.154687+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:22:50.154687+0000","last_undegraded":"2023-10-21T05:22:50.154687+0000","last_fullsized":"2023-10-21T05:22:50.154687+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8345,"log_dups_size":0,"ondisk_log_size":8345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1707","reported_seq":2807,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:45.033568+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:22:45.033568+0000","last_peered":"2023-10-21T05:22:45.033568+0000","last_clean":"2023-10-21T05:22:45.033568+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:22:45.033568+0000","last_undegraded":"2023-10-21T05:22:45.033568+0000","last_fullsized":"2023-10-21T05:22:45.033568+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1707,"log_dups_size":0,"ondisk_log_size":1707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1721","reported_seq":2942,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:45.033644+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:22:45.033644+0000","last_peered":"2023-10-21T05:22:45.033644+0000","last_clean":"2023-10-21T05:22:45.033644+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:22:45.033644+0000","last_undegraded":"2023-10-21T05:22:45.033644+0000","last_fullsized":"2023-10-21T05:22:45.033644+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1721,"log_dups_size":0,"ondisk_log_size":1721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'255853","reported_seq":513993,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:53.448236+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:22:53.448236+0000","last_peered":"2023-10-21T05:22:53.448236+0000","last_clean":"2023-10-21T05:22:53.448236+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:22:53.448236+0000","last_undegraded":"2023-10-21T05:22:53.448236+0000","last_fullsized":"2023-10-21T05:22:53.448236+0000","mapping_epoch":17,"log_start":"54'245821","ondisk_log_start":"54'245821","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523464192,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":257822,"num_read_kb":1004244,"num_write":255853,"num_write_kb":1023412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1771","reported_seq":2880,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:50.184143+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:22:50.184143+0000","last_peered":"2023-10-21T05:22:50.184143+0000","last_clean":"2023-10-21T05:22:50.184143+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:22:50.184143+0000","last_undegraded":"2023-10-21T05:22:50.184143+0000","last_fullsized":"2023-10-21T05:22:50.184143+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1771,"log_dups_size":0,"ondisk_log_size":1771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'239026","reported_seq":481097,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:52.060449+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:22:52.060449+0000","last_peered":"2023-10-21T05:22:52.060449+0000","last_clean":"2023-10-21T05:22:52.060449+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:22:52.060449+0000","last_undegraded":"2023-10-21T05:22:52.060449+0000","last_fullsized":"2023-10-21T05:22:52.060449+0000","mapping_epoch":17,"log_start":"54'228983","ondisk_log_start":"54'228983","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5191192576,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":241765,"num_read_kb":941324,"num_write":239026,"num_write_kb":956104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1110,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:45.033536+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:22:45.033536+0000","last_peered":"2023-10-21T05:22:45.033536+0000","last_clean":"2023-10-21T05:22:45.033536+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:22:45.033536+0000","last_undegraded":"2023-10-21T05:22:45.033536+0000","last_fullsized":"2023-10-21T05:22:45.033536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'248516","reported_seq":499764,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:53.452403+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:22:53.452403+0000","last_peered":"2023-10-21T05:22:53.452403+0000","last_clean":"2023-10-21T05:22:53.452403+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:22:53.452403+0000","last_undegraded":"2023-10-21T05:22:53.452403+0000","last_fullsized":"2023-10-21T05:22:53.452403+0000","mapping_epoch":17,"log_start":"54'238416","ondisk_log_start":"54'238416","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386551296,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":250942,"num_read_kb":977912,"num_write":248516,"num_write_kb":994064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4116,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:50.154938+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:22:50.154938+0000","last_peered":"2023-10-21T05:22:50.154938+0000","last_clean":"2023-10-21T05:22:50.154938+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:22:50.154938+0000","last_undegraded":"2023-10-21T05:22:50.154938+0000","last_fullsized":"2023-10-21T05:22:50.154938+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'242601","reported_seq":488907,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:52.094624+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:22:52.094624+0000","last_peered":"2023-10-21T05:22:52.094624+0000","last_clean":"2023-10-21T05:22:52.094624+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:22:52.094624+0000","last_undegraded":"2023-10-21T05:22:52.094624+0000","last_fullsized":"2023-10-21T05:22:52.094624+0000","mapping_epoch":17,"log_start":"54'232583","ondisk_log_start":"54'232583","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":866,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 307s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261729811,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":246046,"num_read_kb":958556,"num_write":242601,"num_write_kb":970398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6633","reported_seq":10640,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:50.154237+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:22:50.154237+0000","last_peered":"2023-10-21T05:22:50.154237+0000","last_clean":"2023-10-21T05:22:50.154237+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:22:50.154237+0000","last_undegraded":"2023-10-21T05:22:50.154237+0000","last_fullsized":"2023-10-21T05:22:50.154237+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6633,"log_dups_size":0,"ondisk_log_size":6633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":229,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:05.382421+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:20:05.382421+0000","last_peered":"2023-10-21T05:20:05.382421+0000","last_clean":"2023-10-21T05:20:05.382421+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:20:05.382421+0000","last_undegraded":"2023-10-21T05:20:05.382421+0000","last_fullsized":"2023-10-21T05:20:05.382421+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21362937875,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":996575,"num_read_kb":3882036,"num_write":985996,"num_write_kb":3943978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32058630144,"data_stored":32058630144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40193,"ondisk_log_size":40193,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":25304,"ondisk_log_size":25304,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739300,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12207072,"kb_used_data":10436248,"kb_used_omap":8,"kb_used_meta":1770807,"kb_avail":82164768,"statfs":{"total":96636764160,"available":84136722432,"internally_reserved":0,"allocated":10686717952,"data_stored":10686413421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1813307037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.0920000000000001,"15min":0.89800000000000002},"min":{"1min":0.20699999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.7229999999999999,"5min":11.904999999999999,"15min":22.969000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.73399999999999999,"15min":0.73299999999999998},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.17899999999999999},"max":{"1min":4.3780000000000001,"5min":12.252000000000001,"15min":22.835000000000001},"last":0.34899999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:22:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.80100000000000005,"15min":0.70099999999999996},"min":{"1min":0.21299999999999999,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":5.2619999999999996,"5min":6.5199999999999996,"15min":19.574000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.99299999999999999,"15min":0.91800000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":5.4539999999999997,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.31900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739298,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12227312,"kb_used_data":10436904,"kb_used_omap":15,"kb_used_meta":1790384,"kb_avail":82144528,"statfs":{"total":96636764160,"available":84115996672,"internally_reserved":0,"allocated":10687389696,"data_stored":10686971900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1833353606},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,4,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.94499999999999995,"15min":0.85499999999999998},"min":{"1min":0.155,"5min":0.155,"15min":0.13},"max":{"1min":3.4340000000000002,"5min":13.257,"15min":14.682},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.71499999999999997,"15min":0.55100000000000005},"min":{"1min":0.19900000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":9.1820000000000004,"5min":9.1820000000000004,"15min":9.1820000000000004},"last":0.29099999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.65900000000000003,"15min":0.71599999999999997},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.185},"max":{"1min":4.8719999999999999,"5min":14.138,"15min":14.138},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.70099999999999996,"15min":0.72899999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.20599999999999999},"max":{"1min":4.907,"5min":14.173,"15min":14.173},"last":0.443}]}]},{"osd":0,"up_from":8,"seq":34359739300,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12213276,"kb_used_data":10436864,"kb_used_omap":12,"kb_used_meta":1776371,"kb_avail":82158564,"statfs":{"total":96636764160,"available":84130369536,"internally_reserved":0,"allocated":10687348736,"data_stored":10686983739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1819004329},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,8,7,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:22:36 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.50800000000000001,"15min":0.54700000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.17699999999999999},"max":{"1min":5.3209999999999997,"5min":5.3330000000000002,"15min":6.5759999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.67500000000000004,"15min":0.65000000000000002},"min":{"1min":0.23699999999999999,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":5.4059999999999997,"5min":5.4690000000000003,"15min":6.6429999999999998},"last":0.39500000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:22:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.91800000000000004,"15min":0.90500000000000003},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.23499999999999999},"max":{"1min":5.3609999999999998,"5min":6.5190000000000001,"15min":9.3140000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.79400000000000004,"15min":0.89700000000000002},"min":{"1min":0.22800000000000001,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":3.1190000000000002,"5min":6.5549999999999997,"15min":9.516},"last":2.637}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10686201856,"data_stored":10686201856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10686181376,"data_stored":10686181376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10686246912,"data_stored":10686246912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:22:57.293 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:22:57.294 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:23:17.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:23:18.053 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:23:18.054 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:23:18.126 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2396,"stamp":"2023-10-21T05:23:17.823864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21363910499,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":1006672,"num_read_kb":3904059,"num_write":990855,"num_write_kb":3963911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65798,"ondisk_log_size":65798,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":36742696,"kb_used_data":31310736,"kb_used_omap":36,"kb_used_meta":5431899,"kb_avail":246372824,"statfs":{"total":289910292480,"available":252285771776,"internally_reserved":0,"allocated":32062193664,"data_stored":32061106340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":5562265036},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,6,10,17,11,21,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":192512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2491,"num_read_kb":9856,"num_write":2502,"num_write_kb":10008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002135"},"pg_stats":[{"pgid":"2.7","version":"54'5055","reported_seq":8127,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:15.786326+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:23:15.786326+0000","last_peered":"2023-10-21T05:23:15.786326+0000","last_clean":"2023-10-21T05:23:15.786326+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:23:15.786326+0000","last_undegraded":"2023-10-21T05:23:15.786326+0000","last_fullsized":"2023-10-21T05:23:15.786326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8375","reported_seq":13646,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:15.810890+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:23:15.810890+0000","last_peered":"2023-10-21T05:23:15.810890+0000","last_clean":"2023-10-21T05:23:15.810890+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:23:15.810890+0000","last_undegraded":"2023-10-21T05:23:15.810890+0000","last_fullsized":"2023-10-21T05:23:15.810890+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8375,"log_dups_size":0,"ondisk_log_size":8375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1713","reported_seq":2819,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:15.784300+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:23:15.784300+0000","last_peered":"2023-10-21T05:23:15.784300+0000","last_clean":"2023-10-21T05:23:15.784300+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:23:15.784300+0000","last_undegraded":"2023-10-21T05:23:15.784300+0000","last_fullsized":"2023-10-21T05:23:15.784300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1713,"log_dups_size":0,"ondisk_log_size":1713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1727","reported_seq":2954,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:15.760341+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:23:15.760341+0000","last_peered":"2023-10-21T05:23:15.760341+0000","last_clean":"2023-10-21T05:23:15.760341+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:23:15.760341+0000","last_undegraded":"2023-10-21T05:23:15.760341+0000","last_fullsized":"2023-10-21T05:23:15.760341+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1727,"log_dups_size":0,"ondisk_log_size":1727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'256889","reported_seq":516012,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:13.682060+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:23:13.682060+0000","last_peered":"2023-10-21T05:23:13.682060+0000","last_clean":"2023-10-21T05:23:13.682060+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:23:13.682060+0000","last_undegraded":"2023-10-21T05:23:13.682060+0000","last_fullsized":"2023-10-21T05:23:13.682060+0000","mapping_epoch":17,"log_start":"54'246828","ondisk_log_start":"54'246828","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523554304,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":258803,"num_read_kb":1008116,"num_write":256889,"num_write_kb":1027556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1778","reported_seq":2891,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:09.888517+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:23:09.888517+0000","last_peered":"2023-10-21T05:23:09.888517+0000","last_clean":"2023-10-21T05:23:09.888517+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:23:09.888517+0000","last_undegraded":"2023-10-21T05:23:09.888517+0000","last_fullsized":"2023-10-21T05:23:09.888517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1778,"log_dups_size":0,"ondisk_log_size":1778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'240268","reported_seq":483512,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:17.388327+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:23:17.388327+0000","last_peered":"2023-10-21T05:23:17.388327+0000","last_clean":"2023-10-21T05:23:17.388327+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:23:17.388327+0000","last_undegraded":"2023-10-21T05:23:17.388327+0000","last_fullsized":"2023-10-21T05:23:17.388327+0000","mapping_epoch":17,"log_start":"54'230195","ondisk_log_start":"54'230195","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5191266304,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":242939,"num_read_kb":945972,"num_write":240268,"num_write_kb":961072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1114,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:05.513660+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:23:05.513660+0000","last_peered":"2023-10-21T05:23:05.513660+0000","last_clean":"2023-10-21T05:23:05.513660+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:23:05.513660+0000","last_undegraded":"2023-10-21T05:23:05.513660+0000","last_fullsized":"2023-10-21T05:23:05.513660+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'249513","reported_seq":501719,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:13.661794+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:23:13.661794+0000","last_peered":"2023-10-21T05:23:13.661794+0000","last_clean":"2023-10-21T05:23:13.661794+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:23:13.661794+0000","last_undegraded":"2023-10-21T05:23:13.661794+0000","last_fullsized":"2023-10-21T05:23:13.661794+0000","mapping_epoch":17,"log_start":"54'239426","ondisk_log_start":"54'239426","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386661888,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":251901,"num_read_kb":981700,"num_write":249513,"num_write_kb":998052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4132,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:10.633856+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:23:10.633856+0000","last_peered":"2023-10-21T05:23:10.633856+0000","last_clean":"2023-10-21T05:23:10.633856+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:23:10.633856+0000","last_undegraded":"2023-10-21T05:23:10.633856+0000","last_fullsized":"2023-10-21T05:23:10.633856+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'243785","reported_seq":491316,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:17.427922+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:23:17.427922+0000","last_peered":"2023-10-21T05:23:17.427922+0000","last_clean":"2023-10-21T05:23:17.427922+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:23:17.427922+0000","last_undegraded":"2023-10-21T05:23:17.427922+0000","last_fullsized":"2023-10-21T05:23:17.427922+0000","mapping_epoch":17,"log_start":"54'233702","ondisk_log_start":"54'233702","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":932,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 333s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261836307,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":247263,"num_read_kb":963364,"num_write":243785,"num_write_kb":975134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6657","reported_seq":10685,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:15.850810+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:23:15.850810+0000","last_peered":"2023-10-21T05:23:15.850810+0000","last_clean":"2023-10-21T05:23:15.850810+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:23:15.850810+0000","last_undegraded":"2023-10-21T05:23:15.850810+0000","last_fullsized":"2023-10-21T05:23:15.850810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6657,"log_dups_size":0,"ondisk_log_size":6657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":229,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:05.382421+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:20:05.382421+0000","last_peered":"2023-10-21T05:20:05.382421+0000","last_clean":"2023-10-21T05:20:05.382421+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:20:05.382421+0000","last_undegraded":"2023-10-21T05:20:05.382421+0000","last_fullsized":"2023-10-21T05:20:05.382421+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21363318803,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":1000906,"num_read_kb":3899152,"num_write":990455,"num_write_kb":3961814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32059432960,"data_stored":32059432960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40304,"ondisk_log_size":40304,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":25395,"ondisk_log_size":25395,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739304,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12243380,"kb_used_data":10436460,"kb_used_omap":8,"kb_used_meta":1806903,"kb_avail":82128460,"statfs":{"total":96636764160,"available":84099543040,"internally_reserved":0,"allocated":10686935040,"data_stored":10686630509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1850269341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.82899999999999996,"15min":0.875},"min":{"1min":0.184,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.5209999999999999,"5min":6.0270000000000001,"15min":22.969000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.58999999999999997,"15min":0.70899999999999996},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":2.7130000000000001,"5min":5.8689999999999998,"15min":22.835000000000001},"last":0.44700000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:22:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.80100000000000005,"15min":0.70099999999999996},"min":{"1min":0.21299999999999999,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":5.2619999999999996,"5min":6.5199999999999996,"15min":19.574000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.99299999999999999,"15min":0.91800000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":5.4539999999999997,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.54800000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739303,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12267780,"kb_used_data":10437180,"kb_used_omap":15,"kb_used_meta":1830576,"kb_avail":82104060,"statfs":{"total":96636764160,"available":84074557440,"internally_reserved":0,"allocated":10687672320,"data_stored":10687254524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1874510214},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,6,8,17,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.94499999999999995,"15min":0.85499999999999998},"min":{"1min":0.155,"5min":0.155,"15min":0.13},"max":{"1min":3.4340000000000002,"5min":13.257,"15min":14.682},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.71499999999999997,"15min":0.55100000000000005},"min":{"1min":0.19900000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":9.1820000000000004,"5min":9.1820000000000004,"15min":9.1820000000000004},"last":0.40300000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.65900000000000003,"15min":0.71599999999999997},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.185},"max":{"1min":4.8719999999999999,"5min":14.138,"15min":14.138},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.70099999999999996,"15min":0.72899999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.20599999999999999},"max":{"1min":4.907,"5min":14.173,"15min":14.173},"last":0.36699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739304,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12231536,"kb_used_data":10437096,"kb_used_omap":12,"kb_used_meta":1794419,"kb_avail":82140304,"statfs":{"total":96636764160,"available":84111671296,"internally_reserved":0,"allocated":10687586304,"data_stored":10687221307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1837485481},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,9,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:22:36 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.50800000000000001,"15min":0.54700000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.17699999999999999},"max":{"1min":5.3209999999999997,"5min":5.3330000000000002,"15min":6.5759999999999996},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.67500000000000004,"15min":0.65000000000000002},"min":{"1min":0.23699999999999999,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":5.4059999999999997,"5min":5.4690000000000003,"15min":6.6429999999999998},"last":0.26200000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:22:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.91800000000000004,"15min":0.90500000000000003},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.23499999999999999},"max":{"1min":5.3609999999999998,"5min":6.5190000000000001,"15min":9.3140000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.79400000000000004,"15min":0.89700000000000002},"min":{"1min":0.22800000000000001,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":3.1190000000000002,"5min":6.5549999999999997,"15min":9.516},"last":0.34399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10686472192,"data_stored":10686472192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10686484480,"data_stored":10686484480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10686476288,"data_stored":10686476288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:23:18.129 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:23:18.129 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:23:38.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:23:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:23:38.944 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:23:38.999 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2406,"stamp":"2023-10-21T05:23:37.827281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21364172643,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":1010207,"num_read_kb":3918051,"num_write":994333,"num_write_kb":3977823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65811,"ondisk_log_size":65811,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":36169348,"kb_used_data":31311148,"kb_used_omap":36,"kb_used_meta":4858139,"kb_avail":246946172,"statfs":{"total":289910292480,"available":252872880128,"internally_reserved":0,"allocated":32062615552,"data_stored":32061528228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37428,"internal_metadata":4974734796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,14,5,17,11,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":184320,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2143,"num_read_kb":8492,"num_write":2109,"num_write_kb":8436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002003"},"pg_stats":[{"pgid":"2.7","version":"54'5073","reported_seq":8157,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:36.237959+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:23:36.237959+0000","last_peered":"2023-10-21T05:23:36.237959+0000","last_clean":"2023-10-21T05:23:36.237959+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:23:36.237959+0000","last_undegraded":"2023-10-21T05:23:36.237959+0000","last_fullsized":"2023-10-21T05:23:36.237959+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8405","reported_seq":13696,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:36.238520+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:23:36.238520+0000","last_peered":"2023-10-21T05:23:36.238520+0000","last_clean":"2023-10-21T05:23:36.238520+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:23:36.238520+0000","last_undegraded":"2023-10-21T05:23:36.238520+0000","last_fullsized":"2023-10-21T05:23:36.238520+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1719","reported_seq":2829,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:35.062349+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:23:35.062349+0000","last_peered":"2023-10-21T05:23:35.062349+0000","last_clean":"2023-10-21T05:23:35.062349+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:23:35.062349+0000","last_undegraded":"2023-10-21T05:23:35.062349+0000","last_fullsized":"2023-10-21T05:23:35.062349+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1719,"log_dups_size":0,"ondisk_log_size":1719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1733","reported_seq":2964,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:35.069139+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:23:35.069139+0000","last_peered":"2023-10-21T05:23:35.069139+0000","last_clean":"2023-10-21T05:23:35.069139+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:23:35.069139+0000","last_undegraded":"2023-10-21T05:23:35.069139+0000","last_fullsized":"2023-10-21T05:23:35.069139+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1733,"log_dups_size":0,"ondisk_log_size":1733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'257826","reported_seq":517854,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:33.931392+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:23:33.931392+0000","last_peered":"2023-10-21T05:23:33.931392+0000","last_clean":"2023-10-21T05:23:33.931392+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:23:33.931392+0000","last_undegraded":"2023-10-21T05:23:33.931392+0000","last_fullsized":"2023-10-21T05:23:33.931392+0000","mapping_epoch":17,"log_start":"54'247737","ondisk_log_start":"54'247737","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523677184,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":259709,"num_read_kb":1011712,"num_write":257826,"num_write_kb":1031304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1783","reported_seq":2900,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:31.513528+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:23:31.513528+0000","last_peered":"2023-10-21T05:23:31.513528+0000","last_clean":"2023-10-21T05:23:31.513528+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:23:31.513528+0000","last_undegraded":"2023-10-21T05:23:31.513528+0000","last_fullsized":"2023-10-21T05:23:31.513528+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1783,"log_dups_size":0,"ondisk_log_size":1783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'241085","reported_seq":485178,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:37.663710+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:23:37.663710+0000","last_peered":"2023-10-21T05:23:37.663710+0000","last_clean":"2023-10-21T05:23:37.663710+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:23:37.663710+0000","last_undegraded":"2023-10-21T05:23:37.663710+0000","last_fullsized":"2023-10-21T05:23:37.663710+0000","mapping_epoch":17,"log_start":"54'231011","ondisk_log_start":"54'231011","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5191282688,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":243786,"num_read_kb":949332,"num_write":241085,"num_write_kb":964340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1118,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:26.120845+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:23:26.120845+0000","last_peered":"2023-10-21T05:23:26.120845+0000","last_clean":"2023-10-21T05:23:26.120845+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:23:26.120845+0000","last_undegraded":"2023-10-21T05:23:26.120845+0000","last_fullsized":"2023-10-21T05:23:26.120845+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'250369","reported_seq":503488,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:33.911885+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:23:33.911885+0000","last_peered":"2023-10-21T05:23:33.911885+0000","last_clean":"2023-10-21T05:23:33.911885+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:23:33.911885+0000","last_undegraded":"2023-10-21T05:23:33.911885+0000","last_fullsized":"2023-10-21T05:23:33.911885+0000","mapping_epoch":17,"log_start":"54'240337","ondisk_log_start":"54'240337","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386723328,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":252814,"num_read_kb":985304,"num_write":250369,"num_write_kb":1001476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4148,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:31.117705+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:23:31.117705+0000","last_peered":"2023-10-21T05:23:31.117705+0000","last_clean":"2023-10-21T05:23:31.117705+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:23:31.117705+0000","last_undegraded":"2023-10-21T05:23:31.117705+0000","last_fullsized":"2023-10-21T05:23:31.117705+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'244653","reported_seq":493060,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:37.709518+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:23:37.709518+0000","last_peered":"2023-10-21T05:23:37.709518+0000","last_clean":"2023-10-21T05:23:37.709518+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:23:37.709518+0000","last_undegraded":"2023-10-21T05:23:37.709518+0000","last_fullsized":"2023-10-21T05:23:37.709518+0000","mapping_epoch":17,"log_start":"54'234620","ondisk_log_start":"54'234620","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":962,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 353s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261897747,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":248132,"num_read_kb":966796,"num_write":244653,"num_write_kb":978606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6681","reported_seq":10725,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:36.238114+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:23:36.238114+0000","last_peered":"2023-10-21T05:23:36.238114+0000","last_clean":"2023-10-21T05:23:36.238114+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:23:36.238114+0000","last_undegraded":"2023-10-21T05:23:36.238114+0000","last_fullsized":"2023-10-21T05:23:36.238114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6681,"log_dups_size":0,"ondisk_log_size":6681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":229,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:05.382421+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:20:05.382421+0000","last_peered":"2023-10-21T05:20:05.382421+0000","last_clean":"2023-10-21T05:20:05.382421+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:20:05.382421+0000","last_undegraded":"2023-10-21T05:20:05.382421+0000","last_fullsized":"2023-10-21T05:20:05.382421+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21363580947,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":1004441,"num_read_kb":3913144,"num_write":993933,"num_write_kb":3975726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32059797504,"data_stored":32059797504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40228,"ondisk_log_size":40228,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":25484,"ondisk_log_size":25484,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739308,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12277676,"kb_used_data":10436580,"kb_used_omap":8,"kb_used_meta":1841079,"kb_avail":82094164,"statfs":{"total":96636764160,"available":84064423936,"internally_reserved":0,"allocated":10687057920,"data_stored":10686753389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8547,"internal_metadata":1885265565},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.82899999999999996,"15min":0.875},"min":{"1min":0.184,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.5209999999999999,"5min":6.0270000000000001,"15min":22.969000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.58999999999999997,"15min":0.70899999999999996},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":2.7130000000000001,"5min":5.8689999999999998,"15min":22.835000000000001},"last":0.27400000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:22:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.80100000000000005,"15min":0.70099999999999996},"min":{"1min":0.21299999999999999,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":5.2619999999999996,"5min":6.5199999999999996,"15min":19.574000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.99299999999999999,"15min":0.91800000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":5.4539999999999997,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.31900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739307,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11946308,"kb_used_data":10437308,"kb_used_omap":15,"kb_used_meta":1508976,"kb_avail":82425532,"statfs":{"total":96636764160,"available":84403744768,"internally_reserved":0,"allocated":10687803392,"data_stored":10687385596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1545191814},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,0,7,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.94499999999999995,"15min":0.85499999999999998},"min":{"1min":0.155,"5min":0.155,"15min":0.13},"max":{"1min":3.4340000000000002,"5min":13.257,"15min":14.682},"last":1.573},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.71499999999999997,"15min":0.55100000000000005},"min":{"1min":0.19900000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":9.1820000000000004,"5min":9.1820000000000004,"15min":9.1820000000000004},"last":0.25900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.65900000000000003,"15min":0.71599999999999997},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.185},"max":{"1min":4.8719999999999999,"5min":14.138,"15min":14.138},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.70099999999999996,"15min":0.72899999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.20599999999999999},"max":{"1min":4.907,"5min":14.173,"15min":14.173},"last":0.47899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739308,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11945364,"kb_used_data":10437260,"kb_used_omap":12,"kb_used_meta":1508083,"kb_avail":82426476,"statfs":{"total":96636764160,"available":84404711424,"internally_reserved":0,"allocated":10687754240,"data_stored":10687389243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1544277417},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,3,4,7,3,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:22:36 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.50800000000000001,"15min":0.54700000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.17699999999999999},"max":{"1min":5.3209999999999997,"5min":5.3330000000000002,"15min":6.5759999999999996},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.67500000000000004,"15min":0.65000000000000002},"min":{"1min":0.23699999999999999,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":5.4059999999999997,"5min":5.4690000000000003,"15min":6.6429999999999998},"last":0.25800000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:22:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.91800000000000004,"15min":0.90500000000000003},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.23499999999999999},"max":{"1min":5.3609999999999998,"5min":6.5190000000000001,"15min":9.3140000000000001},"last":0.499},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.79400000000000004,"15min":0.89700000000000002},"min":{"1min":0.22800000000000001,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":3.1190000000000002,"5min":6.5549999999999997,"15min":9.516},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10686590976,"data_stored":10686590976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10686623744,"data_stored":10686623744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10686582784,"data_stored":10686582784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:23:39.000 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:23:39.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:23:39.767 INFO:teuthology.orchestra.run.smithi172.stderr:instructing pg 3.2s0 on osd.1 to deep-scrub 2023-10-21T05:23:39.821 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:23:59.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:24:00.394 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:24:00.395 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:24:00.481 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2417,"stamp":"2023-10-21T05:23:59.832338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21364418403,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":1014237,"num_read_kb":3933991,"num_write":998617,"num_write_kb":3994959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65830,"ondisk_log_size":65830,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35079012,"kb_used_data":31311692,"kb_used_omap":38,"kb_used_meta":3767257,"kb_avail":248036508,"statfs":{"total":289910292480,"available":253989384192,"internally_reserved":0,"allocated":32063172608,"data_stored":32062085284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39108,"internal_metadata":3857671996},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,6,14,23,15,22,17,13,10],"upper_bound":1024},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":192512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2749,"num_read_kb":10884,"num_write":2894,"num_write_kb":11576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003214"},"pg_stats":[{"pgid":"2.7","version":"54'5091","reported_seq":8188,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:56.714287+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:23:56.714287+0000","last_peered":"2023-10-21T05:23:56.714287+0000","last_clean":"2023-10-21T05:23:56.714287+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:23:56.714287+0000","last_undegraded":"2023-10-21T05:23:56.714287+0000","last_fullsized":"2023-10-21T05:23:56.714287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8435","reported_seq":13747,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:56.714496+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:23:56.714496+0000","last_peered":"2023-10-21T05:23:56.714496+0000","last_clean":"2023-10-21T05:23:56.714496+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:23:56.714496+0000","last_undegraded":"2023-10-21T05:23:56.714496+0000","last_fullsized":"2023-10-21T05:23:56.714496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8435,"log_dups_size":0,"ondisk_log_size":8435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1725","reported_seq":2839,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:51.593407+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:23:51.593407+0000","last_peered":"2023-10-21T05:23:51.593407+0000","last_clean":"2023-10-21T05:23:51.593407+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:23:51.593407+0000","last_undegraded":"2023-10-21T05:23:51.593407+0000","last_fullsized":"2023-10-21T05:23:51.593407+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1725,"log_dups_size":0,"ondisk_log_size":1725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1739","reported_seq":2974,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:51.593747+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:23:51.593747+0000","last_peered":"2023-10-21T05:23:51.593747+0000","last_clean":"2023-10-21T05:23:51.593747+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:23:51.593747+0000","last_undegraded":"2023-10-21T05:23:51.593747+0000","last_fullsized":"2023-10-21T05:23:51.593747+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1739,"log_dups_size":0,"ondisk_log_size":1739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'259037","reported_seq":520188,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:59.201053+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:23:59.201053+0000","last_peered":"2023-10-21T05:23:59.201053+0000","last_clean":"2023-10-21T05:23:59.201053+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:23:59.201053+0000","last_undegraded":"2023-10-21T05:23:59.201053+0000","last_fullsized":"2023-10-21T05:23:59.201053+0000","mapping_epoch":17,"log_start":"54'248961","ondisk_log_start":"54'248961","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523783680,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":260831,"num_read_kb":1016136,"num_write":259037,"num_write_kb":1036148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1792","reported_seq":2914,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:53.510098+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:23:53.510098+0000","last_peered":"2023-10-21T05:23:53.510098+0000","last_clean":"2023-10-21T05:23:53.510098+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:23:53.510098+0000","last_undegraded":"2023-10-21T05:23:53.510098+0000","last_fullsized":"2023-10-21T05:23:53.510098+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1792,"log_dups_size":0,"ondisk_log_size":1792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'242025","reported_seq":486990,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:57.915806+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:23:57.915806+0000","last_peered":"2023-10-21T05:23:57.915806+0000","last_clean":"2023-10-21T05:23:57.915806+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:23:57.915806+0000","last_undegraded":"2023-10-21T05:23:57.915806+0000","last_fullsized":"2023-10-21T05:23:57.915806+0000","mapping_epoch":17,"log_start":"54'232021","ondisk_log_start":"54'232021","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5191315456,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":244660,"num_read_kb":952788,"num_write":242025,"num_write_kb":968100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1123,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:51.593835+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:23:51.593835+0000","last_peered":"2023-10-21T05:23:51.593835+0000","last_clean":"2023-10-21T05:23:51.593835+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:23:51.593835+0000","last_undegraded":"2023-10-21T05:23:51.593835+0000","last_fullsized":"2023-10-21T05:23:51.593835+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'251588","reported_seq":505858,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:59.201898+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:23:59.201898+0000","last_peered":"2023-10-21T05:23:59.201898+0000","last_clean":"2023-10-21T05:23:59.201898+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:23:59.201898+0000","last_undegraded":"2023-10-21T05:23:59.201898+0000","last_fullsized":"2023-10-21T05:23:59.201898+0000","mapping_epoch":17,"log_start":"54'241548","ondisk_log_start":"54'241548","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386788864,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":253966,"num_read_kb":989852,"num_write":251588,"num_write_kb":1006352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4169,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:56.717259+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:23:56.717259+0000","last_peered":"2023-10-21T05:23:56.717259+0000","last_clean":"2023-10-21T05:23:56.717259+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:23:56.717259+0000","last_undegraded":"2023-10-21T05:23:56.717259+0000","last_fullsized":"2023-10-21T05:23:56.717259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'245567","reported_seq":494870,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:57.886463+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:23:57.886463+0000","last_peered":"2023-10-21T05:23:57.886463+0000","last_clean":"2023-10-21T05:23:57.886463+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:23:57.886463+0000","last_undegraded":"2023-10-21T05:23:57.886463+0000","last_fullsized":"2023-10-21T05:23:57.886463+0000","mapping_epoch":17,"log_start":"54'235533","ondisk_log_start":"54'235533","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1051,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 373s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261938707,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":249014,"num_read_kb":970308,"num_write":245567,"num_write_kb":982262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6705","reported_seq":10765,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:23:56.714510+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:23:56.714510+0000","last_peered":"2023-10-21T05:23:56.714510+0000","last_clean":"2023-10-21T05:23:56.714510+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:23:56.714510+0000","last_undegraded":"2023-10-21T05:23:56.714510+0000","last_fullsized":"2023-10-21T05:23:56.714510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6705,"log_dups_size":0,"ondisk_log_size":6705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":229,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:20:05.382421+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:20:05.382421+0000","last_peered":"2023-10-21T05:20:05.382421+0000","last_clean":"2023-10-21T05:20:05.382421+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:20:05.382421+0000","last_undegraded":"2023-10-21T05:20:05.382421+0000","last_fullsized":"2023-10-21T05:20:05.382421+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21363826707,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":1008471,"num_read_kb":3929084,"num_write":998217,"num_write_kb":3992862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32060243968,"data_stored":32060243968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40154,"ondisk_log_size":40154,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":25577,"ondisk_log_size":25577,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739313,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11163396,"kb_used_data":10436796,"kb_used_omap":9,"kb_used_meta":726582,"kb_avail":83208444,"statfs":{"total":96636764160,"available":85205446656,"internally_reserved":0,"allocated":10687279104,"data_stored":10686974573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10227,"internal_metadata":744019981},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:57 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":0.999,"15min":0.88},"min":{"1min":0.184,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":8.0340000000000007,"5min":8.0340000000000007,"15min":11.904999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.1180000000000001,"5min":0.73999999999999999,"15min":0.68500000000000005},"min":{"1min":0.21299999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":5.7690000000000001,"5min":5.8689999999999998,"15min":12.252000000000001},"last":0.79700000000000004}]},{"osd":1,"last update":"Sat Oct 21 05:23:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.76600000000000001,"15min":0.70099999999999996},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.6520000000000001,"5min":6.5199999999999996,"15min":19.574000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.98899999999999999,"15min":0.90300000000000002},"min":{"1min":0.254,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":4.0019999999999998,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.76200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739311,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11964516,"kb_used_data":10437468,"kb_used_omap":15,"kb_used_meta":1527024,"kb_avail":82407324,"statfs":{"total":96636764160,"available":84385099776,"internally_reserved":0,"allocated":10687967232,"data_stored":10687549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1563672966},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,8,10,9,14,11,8,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:42 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":0.95499999999999996,"15min":0.86699999999999999},"min":{"1min":0.185,"5min":0.155,"15min":0.13},"max":{"1min":5.9909999999999997,"5min":10.311999999999999,"15min":14.682},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.70799999999999996,"15min":0.54300000000000004},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.9079999999999999,"5min":9.1820000000000004,"15min":9.1820000000000004},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.72499999999999998,"15min":0.71899999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.185},"max":{"1min":7.0110000000000001,"5min":14.138,"15min":14.138},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.78500000000000003,"15min":0.752},"min":{"1min":0.34999999999999998,"5min":0.28799999999999998,"15min":0.224},"max":{"1min":7.0439999999999996,"5min":14.173,"15min":14.173},"last":0.47599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739313,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11951100,"kb_used_data":10437428,"kb_used_omap":12,"kb_used_meta":1513651,"kb_avail":82420740,"statfs":{"total":96636764160,"available":84398837760,"internally_reserved":0,"allocated":10687926272,"data_stored":10687561275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1549979049},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,5,10,2,7,6,5,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.57299999999999995,"15min":0.55700000000000005},"min":{"1min":0.187,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5759999999999996},"last":0.318},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.70399999999999996,"15min":0.63400000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.552,"5min":5.4690000000000003,"15min":6.6429999999999998},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.92300000000000004,"15min":0.88200000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23499999999999999},"max":{"1min":1.8,"5min":6.5190000000000001,"15min":9.3140000000000001},"last":3.6829999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.81799999999999995,"15min":0.875},"min":{"1min":0.20599999999999999,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":2.637,"5min":6.5549999999999997,"15min":9.516},"last":3.6600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10686742528,"data_stored":10686742528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10686734336,"data_stored":10686734336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10686767104,"data_stored":10686767104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:24:00.483 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:24:00.484 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:24:13.135 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5a54340 2023-10-21T05:24:13.145 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e48f70 2023-10-21T05:24:13.146 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e491e0 2023-10-21T05:24:13.146 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae5e49d40 2023-10-21T05:24:13.146 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae696e5b0 2023-10-21T05:24:13.147 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae6a1a8f0 2023-10-21T05:24:13.147 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f31e0 2023-10-21T05:24:13.147 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae73f3450 2023-10-21T05:24:13.148 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae755c340 2023-10-21T05:24:13.148 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7e3fee0 2023-10-21T05:24:13.148 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae7f9e1a0 2023-10-21T05:24:13.148 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae8303ad0 2023-10-21T05:24:13.149 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9930 2023-10-21T05:24:13.149 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-21T05:24:13.126+0000 7f7ead54a640 -1 reset not still connected to 0x559ae94b9a00 2023-10-21T05:24:20.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:24:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:24:21.299 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:24:21.351 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2427,"stamp":"2023-10-21T05:24:19.835895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21364598627,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":1017968,"num_read_kb":3948771,"num_write":1002552,"num_write_kb":4010699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65993,"ondisk_log_size":65993,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35133412,"kb_used_data":31311948,"kb_used_omap":38,"kb_used_meta":3821401,"kb_avail":247982108,"statfs":{"total":289910292480,"available":253933678592,"internally_reserved":0,"allocated":32063434752,"data_stored":32062347428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39108,"internal_metadata":3913115452},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,1,2,13,6,28,13,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":131072,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2825,"num_read_kb":11184,"num_write":2963,"num_write_kb":11852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002218"},"pg_stats":[{"pgid":"2.7","version":"54'5103","reported_seq":8212,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:17.193518+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:24:17.193518+0000","last_peered":"2023-10-21T05:24:17.193518+0000","last_clean":"2023-10-21T05:24:17.193518+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:24:17.193518+0000","last_undegraded":"2023-10-21T05:24:17.193518+0000","last_fullsized":"2023-10-21T05:24:17.193518+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":5103,"log_dups_size":0,"ondisk_log_size":5103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8455","reported_seq":13787,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:17.194338+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:24:17.194338+0000","last_peered":"2023-10-21T05:24:17.194338+0000","last_clean":"2023-10-21T05:24:17.194338+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:24:17.194338+0000","last_undegraded":"2023-10-21T05:24:17.194338+0000","last_fullsized":"2023-10-21T05:24:17.194338+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8455,"log_dups_size":0,"ondisk_log_size":8455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1729","reported_seq":2847,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:13.093709+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:24:13.093709+0000","last_peered":"2023-10-21T05:24:13.093709+0000","last_clean":"2023-10-21T05:24:13.093709+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:24:13.093709+0000","last_undegraded":"2023-10-21T05:24:13.093709+0000","last_fullsized":"2023-10-21T05:24:13.093709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1729,"log_dups_size":0,"ondisk_log_size":1729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1743","reported_seq":2982,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:13.099616+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:24:13.099616+0000","last_peered":"2023-10-21T05:24:13.099616+0000","last_clean":"2023-10-21T05:24:13.099616+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:24:13.099616+0000","last_undegraded":"2023-10-21T05:24:13.099616+0000","last_fullsized":"2023-10-21T05:24:13.099616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1743,"log_dups_size":0,"ondisk_log_size":1743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'260018","reported_seq":522171,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:19.424245+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:24:19.424245+0000","last_peered":"2023-10-21T05:24:19.424245+0000","last_clean":"2023-10-21T05:24:19.424245+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:24:19.424245+0000","last_undegraded":"2023-10-21T05:24:19.424245+0000","last_fullsized":"2023-10-21T05:24:19.424245+0000","mapping_epoch":17,"log_start":"54'249967","ondisk_log_start":"54'249967","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523812352,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":261834,"num_read_kb":1020112,"num_write":260018,"num_write_kb":1040072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1804","reported_seq":2930,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:15.324360+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:24:15.324360+0000","last_peered":"2023-10-21T05:24:15.324360+0000","last_clean":"2023-10-21T05:24:15.324360+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:24:15.324360+0000","last_undegraded":"2023-10-21T05:24:15.324360+0000","last_fullsized":"2023-10-21T05:24:15.324360+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1804,"log_dups_size":0,"ondisk_log_size":1804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'243018","reported_seq":488831,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:18.139738+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:24:18.139738+0000","last_peered":"2023-10-21T05:24:18.139738+0000","last_clean":"2023-10-21T05:24:18.139738+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:24:18.139738+0000","last_undegraded":"2023-10-21T05:24:18.139738+0000","last_fullsized":"2023-10-21T05:24:18.139738+0000","mapping_epoch":17,"log_start":"54'232931","ondisk_log_start":"54'232931","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5191348224,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":245506,"num_read_kb":956152,"num_write":243018,"num_write_kb":972072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1127,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:12.075044+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:24:12.075044+0000","last_peered":"2023-10-21T05:24:12.075044+0000","last_clean":"2023-10-21T05:24:12.075044+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:24:12.075044+0000","last_undegraded":"2023-10-21T05:24:12.075044+0000","last_fullsized":"2023-10-21T05:24:12.075044+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'252626","reported_seq":507812,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:19.438602+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:24:19.438602+0000","last_peered":"2023-10-21T05:24:19.438602+0000","last_clean":"2023-10-21T05:24:19.438602+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:24:19.438602+0000","last_undegraded":"2023-10-21T05:24:19.438602+0000","last_fullsized":"2023-10-21T05:24:19.438602+0000","mapping_epoch":17,"log_start":"54'242565","ondisk_log_start":"54'242565","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386870784,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":254880,"num_read_kb":993456,"num_write":252626,"num_write_kb":1010504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4185,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:17.233431+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:24:17.233431+0000","last_peered":"2023-10-21T05:24:17.233431+0000","last_clean":"2023-10-21T05:24:17.233431+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:24:17.233431+0000","last_undegraded":"2023-10-21T05:24:17.233431+0000","last_fullsized":"2023-10-21T05:24:17.233431+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'246490","reported_seq":496767,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:18.193118+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:24:18.193118+0000","last_peered":"2023-10-21T05:24:18.193118+0000","last_clean":"2023-10-21T05:24:18.193118+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:24:18.193118+0000","last_undegraded":"2023-10-21T05:24:18.193118+0000","last_fullsized":"2023-10-21T05:24:18.193118+0000","mapping_epoch":17,"log_start":"54'236440","ondisk_log_start":"54'236440","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1081,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 393s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261975571,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":249982,"num_read_kb":974144,"num_write":246490,"num_write_kb":985954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6721","reported_seq":10797,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:17.249539+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:24:17.249539+0000","last_peered":"2023-10-21T05:24:17.249539+0000","last_clean":"2023-10-21T05:24:17.249539+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:24:17.249539+0000","last_undegraded":"2023-10-21T05:24:17.249539+0000","last_fullsized":"2023-10-21T05:24:17.249539+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6721,"log_dups_size":0,"ondisk_log_size":6721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":230,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:06.577736+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:22:06.577736+0000","last_peered":"2023-10-21T05:22:06.577736+0000","last_clean":"2023-10-21T05:22:06.577736+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:22:06.577736+0000","last_undegraded":"2023-10-21T05:22:06.577736+0000","last_fullsized":"2023-10-21T05:22:06.577736+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21364006931,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":1012202,"num_read_kb":3943864,"num_write":1002152,"num_write_kb":4008602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32060583936,"data_stored":32060583936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40249,"ondisk_log_size":40249,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":25645,"ondisk_log_size":25645,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739317,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11181500,"kb_used_data":10436852,"kb_used_omap":9,"kb_used_meta":744630,"kb_avail":83190340,"statfs":{"total":96636764160,"available":85186908160,"internally_reserved":0,"allocated":10687336448,"data_stored":10687031917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10227,"internal_metadata":762501133},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:57 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":0.999,"15min":0.88},"min":{"1min":0.184,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":8.0340000000000007,"5min":8.0340000000000007,"15min":11.904999999999999},"last":0.312},{"interface":"front","average":{"1min":1.1180000000000001,"5min":0.73999999999999999,"15min":0.68500000000000005},"min":{"1min":0.21299999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":5.7690000000000001,"5min":5.8689999999999998,"15min":12.252000000000001},"last":0.28999999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:23:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.76600000000000001,"15min":0.70099999999999996},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.6520000000000001,"5min":6.5199999999999996,"15min":19.574000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.98899999999999999,"15min":0.90300000000000002},"min":{"1min":0.254,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":4.0019999999999998,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.253}]}]},{"osd":1,"up_from":8,"seq":34359739315,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11964632,"kb_used_data":10437584,"kb_used_omap":15,"kb_used_meta":1527024,"kb_avail":82407208,"statfs":{"total":96636764160,"available":84384980992,"internally_reserved":0,"allocated":10688086016,"data_stored":10687668220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1563672966},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,3,1,7,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:42 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":0.95499999999999996,"15min":0.86699999999999999},"min":{"1min":0.185,"5min":0.155,"15min":0.13},"max":{"1min":5.9909999999999997,"5min":10.311999999999999,"15min":14.682},"last":0.192},{"interface":"front","average":{"1min":0.438,"5min":0.70799999999999996,"15min":0.54300000000000004},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.9079999999999999,"5min":9.1820000000000004,"15min":9.1820000000000004},"last":0.22900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.72499999999999998,"15min":0.71899999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.185},"max":{"1min":7.0110000000000001,"5min":14.138,"15min":14.138},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.78500000000000003,"15min":0.752},"min":{"1min":0.34999999999999998,"5min":0.28799999999999998,"15min":0.224},"max":{"1min":7.0439999999999996,"5min":14.173,"15min":14.173},"last":0.374}]}]},{"osd":0,"up_from":8,"seq":34359739317,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11987280,"kb_used_data":10437512,"kb_used_omap":12,"kb_used_meta":1549747,"kb_avail":82384560,"statfs":{"total":96636764160,"available":84361789440,"internally_reserved":0,"allocated":10688012288,"data_stored":10687647291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1586941353},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,7,2,19,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.57299999999999995,"15min":0.55700000000000005},"min":{"1min":0.187,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5759999999999996},"last":0.435},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.70399999999999996,"15min":0.63400000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.552,"5min":5.4690000000000003,"15min":6.6429999999999998},"last":0.51200000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.92300000000000004,"15min":0.88200000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23499999999999999},"max":{"1min":1.8,"5min":6.5190000000000001,"15min":9.3140000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.81799999999999995,"15min":0.875},"min":{"1min":0.20599999999999999,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":2.637,"5min":6.5549999999999997,"15min":9.516},"last":0.20899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10686849024,"data_stored":10686849024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10686873600,"data_stored":10686873600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10686861312,"data_stored":10686861312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:24:21.353 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:24:21.353 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:24:41.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:24:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:24:42.201 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:24:42.257 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2438,"stamp":"2023-10-21T05:24:41.839670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21364721507,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":1021504,"num_read_kb":3962791,"num_write":1006383,"num_write_kb":4026023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66033,"ondisk_log_size":66033,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35044116,"kb_used_data":31312124,"kb_used_omap":38,"kb_used_meta":3731929,"kb_avail":248071404,"statfs":{"total":289910292480,"available":254025117696,"internally_reserved":0,"allocated":32063614976,"data_stored":32062527652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39108,"internal_metadata":3821496124},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,12,9,5,18,26,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":65536,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1736,"num_read_kb":6892,"num_write":1922,"num_write_kb":7688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002067"},"pg_stats":[{"pgid":"2.7","version":"54'5118","reported_seq":8239,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:37.673857+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:24:37.673857+0000","last_peered":"2023-10-21T05:24:37.673857+0000","last_clean":"2023-10-21T05:24:37.673857+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:24:37.673857+0000","last_undegraded":"2023-10-21T05:24:37.673857+0000","last_fullsized":"2023-10-21T05:24:37.673857+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":5118,"log_dups_size":0,"ondisk_log_size":5118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8480","reported_seq":13832,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:37.674526+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:24:37.674526+0000","last_peered":"2023-10-21T05:24:37.674526+0000","last_clean":"2023-10-21T05:24:37.674526+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:24:37.674526+0000","last_undegraded":"2023-10-21T05:24:37.674526+0000","last_fullsized":"2023-10-21T05:24:37.674526+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1734","reported_seq":2856,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:35.733384+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:24:35.733384+0000","last_peered":"2023-10-21T05:24:35.733384+0000","last_clean":"2023-10-21T05:24:35.733384+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:24:35.733384+0000","last_undegraded":"2023-10-21T05:24:35.733384+0000","last_fullsized":"2023-10-21T05:24:35.733384+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1734,"log_dups_size":0,"ondisk_log_size":1734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1748","reported_seq":2991,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:35.733492+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:24:35.733492+0000","last_peered":"2023-10-21T05:24:35.733492+0000","last_clean":"2023-10-21T05:24:35.733492+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:24:35.733492+0000","last_undegraded":"2023-10-21T05:24:35.733492+0000","last_fullsized":"2023-10-21T05:24:35.733492+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1748,"log_dups_size":0,"ondisk_log_size":1748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'260952","reported_seq":524011,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:39.614495+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:24:39.614495+0000","last_peered":"2023-10-21T05:24:39.614495+0000","last_clean":"2023-10-21T05:24:39.614495+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:24:39.614495+0000","last_undegraded":"2023-10-21T05:24:39.614495+0000","last_fullsized":"2023-10-21T05:24:39.614495+0000","mapping_epoch":17,"log_start":"54'250880","ondisk_log_start":"54'250880","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523828736,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":262744,"num_read_kb":1023716,"num_write":260952,"num_write_kb":1043808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1811","reported_seq":2941,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:37.674697+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:24:37.674697+0000","last_peered":"2023-10-21T05:24:37.674697+0000","last_clean":"2023-10-21T05:24:37.674697+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:24:37.674697+0000","last_undegraded":"2023-10-21T05:24:37.674697+0000","last_fullsized":"2023-10-21T05:24:37.674697+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1811,"log_dups_size":0,"ondisk_log_size":1811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'244004","reported_seq":490679,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:38.460938+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:24:38.460938+0000","last_peered":"2023-10-21T05:24:38.460938+0000","last_clean":"2023-10-21T05:24:38.460938+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:24:38.460938+0000","last_undegraded":"2023-10-21T05:24:38.460938+0000","last_fullsized":"2023-10-21T05:24:38.460938+0000","mapping_epoch":17,"log_start":"54'233943","ondisk_log_start":"54'233943","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5191397376,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":246368,"num_read_kb":959564,"num_write":244004,"num_write_kb":976016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1132,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:33.697897+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:24:33.697897+0000","last_peered":"2023-10-21T05:24:33.697897+0000","last_clean":"2023-10-21T05:24:33.697897+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:24:33.697897+0000","last_undegraded":"2023-10-21T05:24:33.697897+0000","last_fullsized":"2023-10-21T05:24:33.697897+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'253653","reported_seq":509729,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:39.595489+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:24:39.595489+0000","last_peered":"2023-10-21T05:24:39.595489+0000","last_clean":"2023-10-21T05:24:39.595489+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:24:39.595489+0000","last_undegraded":"2023-10-21T05:24:39.595489+0000","last_fullsized":"2023-10-21T05:24:39.595489+0000","mapping_epoch":17,"log_start":"54'243589","ondisk_log_start":"54'243589","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5386919936,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":255769,"num_read_kb":996988,"num_write":253653,"num_write_kb":1014612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4201,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:37.685371+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:24:37.685371+0000","last_peered":"2023-10-21T05:24:37.685371+0000","last_clean":"2023-10-21T05:24:37.685371+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:24:37.685371+0000","last_undegraded":"2023-10-21T05:24:37.685371+0000","last_fullsized":"2023-10-21T05:24:37.685371+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'247374","reported_seq":498534,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:38.527105+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:24:38.527105+0000","last_peered":"2023-10-21T05:24:38.527105+0000","last_clean":"2023-10-21T05:24:38.527105+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:24:38.527105+0000","last_undegraded":"2023-10-21T05:24:38.527105+0000","last_fullsized":"2023-10-21T05:24:38.527105+0000","mapping_epoch":17,"log_start":"54'237359","ondisk_log_start":"54'237359","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1147,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 414s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5261983763,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":250857,"num_read_kb":977616,"num_write":247374,"num_write_kb":989490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6741","reported_seq":10833,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:37.674127+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:24:37.674127+0000","last_peered":"2023-10-21T05:24:37.674127+0000","last_clean":"2023-10-21T05:24:37.674127+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:24:37.674127+0000","last_undegraded":"2023-10-21T05:24:37.674127+0000","last_fullsized":"2023-10-21T05:24:37.674127+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6741,"log_dups_size":0,"ondisk_log_size":6741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":230,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:06.577736+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:22:06.577736+0000","last_peered":"2023-10-21T05:22:06.577736+0000","last_clean":"2023-10-21T05:22:06.577736+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:22:06.577736+0000","last_undegraded":"2023-10-21T05:22:06.577736+0000","last_fullsized":"2023-10-21T05:22:06.577736+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21364129811,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":1015738,"num_read_kb":3957884,"num_write":1005983,"num_write_kb":4023926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32060784640,"data_stored":32060784640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40212,"ondisk_log_size":40212,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":25722,"ondisk_log_size":25722,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739321,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11038024,"kb_used_data":10436928,"kb_used_omap":9,"kb_used_meta":601078,"kb_avail":83333816,"statfs":{"total":96636764160,"available":85333827584,"internally_reserved":0,"allocated":10687414272,"data_stored":10687109741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10227,"internal_metadata":615503885},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:57 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":0.999,"15min":0.88},"min":{"1min":0.184,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":8.0340000000000007,"5min":8.0340000000000007,"15min":11.904999999999999},"last":0.19},{"interface":"front","average":{"1min":1.1180000000000001,"5min":0.73999999999999999,"15min":0.68500000000000005},"min":{"1min":0.21299999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":5.7690000000000001,"5min":5.8689999999999998,"15min":12.252000000000001},"last":0.33600000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:23:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.76600000000000001,"15min":0.70099999999999996},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.6520000000000001,"5min":6.5199999999999996,"15min":19.574000000000002},"last":0.223},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.98899999999999999,"15min":0.90300000000000002},"min":{"1min":0.254,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":4.0019999999999998,"5min":8.3840000000000003,"15min":21.234999999999999},"last":0.38100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739319,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12000720,"kb_used_data":10437640,"kb_used_omap":15,"kb_used_meta":1563056,"kb_avail":82371120,"statfs":{"total":96636764160,"available":84348026880,"internally_reserved":0,"allocated":10688143360,"data_stored":10687725564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1600569734},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,3,4,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:42 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":0.95499999999999996,"15min":0.86699999999999999},"min":{"1min":0.185,"5min":0.155,"15min":0.13},"max":{"1min":5.9909999999999997,"5min":10.311999999999999,"15min":14.682},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.70799999999999996,"15min":0.54300000000000004},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.9079999999999999,"5min":9.1820000000000004,"15min":9.1820000000000004},"last":0.309}]},{"osd":2,"last update":"Sat Oct 21 05:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.72499999999999998,"15min":0.71899999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.185},"max":{"1min":7.0110000000000001,"5min":14.138,"15min":14.138},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.78500000000000003,"15min":0.752},"min":{"1min":0.34999999999999998,"5min":0.28799999999999998,"15min":0.224},"max":{"1min":7.0439999999999996,"5min":14.173,"15min":14.173},"last":0.42299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739321,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12005372,"kb_used_data":10437556,"kb_used_omap":12,"kb_used_meta":1567795,"kb_avail":82366468,"statfs":{"total":96636764160,"available":84343263232,"internally_reserved":0,"allocated":10688057344,"data_stored":10687692347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1605422505},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,10,1,2,14,10,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.57299999999999995,"15min":0.55700000000000005},"min":{"1min":0.187,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5759999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.70399999999999996,"15min":0.63400000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.552,"5min":5.4690000000000003,"15min":6.6429999999999998},"last":0.41499999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.92300000000000004,"15min":0.88200000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23499999999999999},"max":{"1min":1.8,"5min":6.5190000000000001,"15min":9.3140000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.81799999999999995,"15min":0.875},"min":{"1min":0.20599999999999999,"5min":0.19700000000000001,"15min":0.17599999999999999},"max":{"1min":2.637,"5min":6.5549999999999997,"15min":9.516},"last":0.33800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10686935040,"data_stored":10686935040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10686939136,"data_stored":10686939136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10686910464,"data_stored":10686910464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:24:42.259 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:24:42.259 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:25:02.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:25:02.893 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:25:02.894 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:25:02.921 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2448,"stamp":"2023-10-21T05:25:01.845506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21365008227,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":1024936,"num_read_kb":3976387,"num_write":1010164,"num_write_kb":4041147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66009,"ondisk_log_size":66009,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35171916,"kb_used_data":31312500,"kb_used_omap":38,"kb_used_meta":3859353,"kb_avail":247943604,"statfs":{"total":289910292480,"available":253894250496,"internally_reserved":0,"allocated":32064000000,"data_stored":32062912676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39108,"internal_metadata":3951978300},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,7,10,10,9,18,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":155648,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1707,"num_read_kb":6752,"num_write":1901,"num_write_kb":7604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004509"},"pg_stats":[{"pgid":"2.7","version":"54'5127","reported_seq":8260,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:58.153642+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:24:58.153642+0000","last_peered":"2023-10-21T05:24:58.153642+0000","last_clean":"2023-10-21T05:24:58.153642+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:24:58.153642+0000","last_undegraded":"2023-10-21T05:24:58.153642+0000","last_fullsized":"2023-10-21T05:24:58.153642+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":5127,"log_dups_size":0,"ondisk_log_size":5127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8495","reported_seq":13867,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:58.154249+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:24:58.154249+0000","last_peered":"2023-10-21T05:24:58.154249+0000","last_clean":"2023-10-21T05:24:58.154249+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:24:58.154249+0000","last_undegraded":"2023-10-21T05:24:58.154249+0000","last_fullsized":"2023-10-21T05:24:58.154249+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8495,"log_dups_size":0,"ondisk_log_size":8495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1737","reported_seq":2863,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:55.970665+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:24:55.970665+0000","last_peered":"2023-10-21T05:24:55.970665+0000","last_clean":"2023-10-21T05:24:55.970665+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:24:55.970665+0000","last_undegraded":"2023-10-21T05:24:55.970665+0000","last_fullsized":"2023-10-21T05:24:55.970665+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1737,"log_dups_size":0,"ondisk_log_size":1737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1751","reported_seq":2998,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:55.952256+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:24:55.952256+0000","last_peered":"2023-10-21T05:24:55.952256+0000","last_clean":"2023-10-21T05:24:55.952256+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:24:55.952256+0000","last_undegraded":"2023-10-21T05:24:55.952256+0000","last_fullsized":"2023-10-21T05:24:55.952256+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1751,"log_dups_size":0,"ondisk_log_size":1751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'261936","reported_seq":525912,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:59.834181+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:24:59.834181+0000","last_peered":"2023-10-21T05:24:59.834181+0000","last_clean":"2023-10-21T05:24:59.834181+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:24:59.834181+0000","last_undegraded":"2023-10-21T05:24:59.834181+0000","last_fullsized":"2023-10-21T05:24:59.834181+0000","mapping_epoch":17,"log_start":"54'251904","ondisk_log_start":"54'251904","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523922944,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":263657,"num_read_kb":1027344,"num_write":261936,"num_write_kb":1047744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1818","reported_seq":2952,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:58.154266+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:24:58.154266+0000","last_peered":"2023-10-21T05:24:58.154266+0000","last_clean":"2023-10-21T05:24:58.154266+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:24:58.154266+0000","last_undegraded":"2023-10-21T05:24:58.154266+0000","last_fullsized":"2023-10-21T05:24:58.154266+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1818,"log_dups_size":0,"ondisk_log_size":1818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'244866","reported_seq":492375,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:58.774188+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:24:58.774188+0000","last_peered":"2023-10-21T05:24:58.774188+0000","last_clean":"2023-10-21T05:24:58.774188+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:24:58.774188+0000","last_undegraded":"2023-10-21T05:24:58.774188+0000","last_fullsized":"2023-10-21T05:24:58.774188+0000","mapping_epoch":17,"log_start":"54'234851","ondisk_log_start":"54'234851","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5191471104,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":247202,"num_read_kb":962852,"num_write":244866,"num_write_kb":979464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1136,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:53.079969+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:24:53.079969+0000","last_peered":"2023-10-21T05:24:53.079969+0000","last_clean":"2023-10-21T05:24:53.079969+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:24:53.079969+0000","last_undegraded":"2023-10-21T05:24:53.079969+0000","last_fullsized":"2023-10-21T05:24:53.079969+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'254627","reported_seq":511571,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:59.821649+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:24:59.821649+0000","last_peered":"2023-10-21T05:24:59.821649+0000","last_clean":"2023-10-21T05:24:59.821649+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:24:59.821649+0000","last_undegraded":"2023-10-21T05:24:59.821649+0000","last_fullsized":"2023-10-21T05:24:59.821649+0000","mapping_epoch":17,"log_start":"54'244601","ondisk_log_start":"54'244601","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5387001856,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":256638,"num_read_kb":1000436,"num_write":254627,"num_write_kb":1018508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4217,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:58.153875+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:24:58.153875+0000","last_peered":"2023-10-21T05:24:58.153875+0000","last_clean":"2023-10-21T05:24:58.153875+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:24:58.153875+0000","last_undegraded":"2023-10-21T05:24:58.153875+0000","last_fullsized":"2023-10-21T05:24:58.153875+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'248335","reported_seq":500320,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:58.752872+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:24:58.752872+0000","last_peered":"2023-10-21T05:24:58.752872+0000","last_clean":"2023-10-21T05:24:58.752872+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:24:58.752872+0000","last_undegraded":"2023-10-21T05:24:58.752872+0000","last_fullsized":"2023-10-21T05:24:58.752872+0000","mapping_epoch":17,"log_start":"54'238269","ondisk_log_start":"54'238269","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1180,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 434s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5262020627,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":251673,"num_read_kb":980848,"num_write":248335,"num_write_kb":993334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6753","reported_seq":10861,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:24:58.154291+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:24:58.154291+0000","last_peered":"2023-10-21T05:24:58.154291+0000","last_clean":"2023-10-21T05:24:58.154291+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:24:58.154291+0000","last_undegraded":"2023-10-21T05:24:58.154291+0000","last_fullsized":"2023-10-21T05:24:58.154291+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6753,"log_dups_size":0,"ondisk_log_size":6753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":230,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:06.577736+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:22:06.577736+0000","last_peered":"2023-10-21T05:22:06.577736+0000","last_clean":"2023-10-21T05:22:06.577736+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:22:06.577736+0000","last_undegraded":"2023-10-21T05:22:06.577736+0000","last_fullsized":"2023-10-21T05:22:06.577736+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21364416531,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":1019170,"num_read_kb":3971480,"num_write":1009764,"num_write_kb":4039050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32061214720,"data_stored":32061214720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40139,"ondisk_log_size":40139,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":25771,"ondisk_log_size":25771,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739325,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11056188,"kb_used_data":10437044,"kb_used_omap":9,"kb_used_meta":619126,"kb_avail":83315652,"statfs":{"total":96636764160,"available":85315227648,"internally_reserved":0,"allocated":10687533056,"data_stored":10687228525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10227,"internal_metadata":633985037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,2,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:57 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":0.999,"15min":0.88},"min":{"1min":0.184,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":8.0340000000000007,"5min":8.0340000000000007,"15min":11.904999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":1.1180000000000001,"5min":0.73999999999999999,"15min":0.68500000000000005},"min":{"1min":0.21299999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":5.7690000000000001,"5min":5.8689999999999998,"15min":12.252000000000001},"last":0.40899999999999997}]},{"osd":1,"last update":"Sat Oct 21 05:24:44 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.72899999999999998,"15min":0.71199999999999997},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.189},"max":{"1min":1.9419999999999999,"5min":6.5199999999999996,"15min":19.574000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.752,"15min":0.88},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19700000000000001},"max":{"1min":1.2529999999999999,"5min":6.8440000000000003,"15min":21.234999999999999},"last":2.6299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739323,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12055796,"kb_used_data":10437740,"kb_used_omap":15,"kb_used_meta":1618032,"kb_avail":82316044,"statfs":{"total":96636764160,"available":84291629056,"internally_reserved":0,"allocated":10688245760,"data_stored":10687827964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1656865158},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,6,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:42 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":0.96399999999999997,"15min":0.876},"min":{"1min":0.192,"5min":0.155,"15min":0.13300000000000001},"max":{"1min":7.9130000000000003,"5min":10.311999999999999,"15min":14.682},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.628,"15min":0.56200000000000006},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.9249999999999998,"5min":9.1820000000000004,"15min":9.1820000000000004},"last":0.28699999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:24:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.71099999999999997,"15min":0.67300000000000004},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":0.44400000000000001,"5min":14.138,"15min":14.138},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.77300000000000002,"15min":0.69399999999999995},"min":{"1min":0.34300000000000003,"5min":0.28799999999999998,"15min":0.22600000000000001},"max":{"1min":0.53700000000000003,"5min":14.173,"15min":14.173},"last":0.441}]}]},{"osd":0,"up_from":8,"seq":34359739325,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12059932,"kb_used_data":10437716,"kb_used_omap":12,"kb_used_meta":1622195,"kb_avail":82311908,"statfs":{"total":96636764160,"available":84287393792,"internally_reserved":0,"allocated":10688221184,"data_stored":10687856187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1661128105},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,5,4,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.51200000000000001,"15min":0.55800000000000005},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":0.76200000000000001,"5min":6.5449999999999999,"15min":6.5759999999999996},"last":0.623},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.56200000000000006,"15min":0.629},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.52700000000000002,"5min":5.4059999999999997,"15min":6.6429999999999998},"last":0.65900000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:24:41 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.0189999999999999,"15min":0.93899999999999995},"min":{"1min":0.27600000000000002,"5min":0.23999999999999999,"15min":0.23499999999999999},"max":{"1min":20.888999999999999,"5min":20.888999999999999,"15min":20.888999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.728,"5min":0.96699999999999997,"15min":0.93899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.17599999999999999},"max":{"1min":22.489999999999998,"5min":22.489999999999998,"15min":22.489999999999998},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10687062016,"data_stored":10687062016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10687070208,"data_stored":10687070208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10687082496,"data_stored":10687082496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:25:02.923 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:25:02.923 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:25:22.924 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:25:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:25:23.509 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:25:23.561 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2458,"stamp":"2023-10-21T05:25:21.849063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21365208931,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":1028588,"num_read_kb":3990911,"num_write":1014081,"num_write_kb":4056815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66186,"ondisk_log_size":66186,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35227652,"kb_used_data":31312680,"kb_used_omap":38,"kb_used_meta":3914905,"kb_avail":247887868,"statfs":{"total":289910292480,"available":253837176832,"internally_reserved":0,"allocated":32064184320,"data_stored":32063096996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39108,"internal_metadata":4008863548},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,3,0,1,11,13,8,8,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":86016,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2303,"num_read_kb":9160,"num_write":2462,"num_write_kb":9848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002143"},"pg_stats":[{"pgid":"2.7","version":"54'5151","reported_seq":8296,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:18.634711+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:25:18.634711+0000","last_peered":"2023-10-21T05:25:18.634711+0000","last_clean":"2023-10-21T05:25:18.634711+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:25:18.634711+0000","last_undegraded":"2023-10-21T05:25:18.634711+0000","last_fullsized":"2023-10-21T05:25:18.634711+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8535","reported_seq":13927,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:18.635654+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:25:18.635654+0000","last_peered":"2023-10-21T05:25:18.635654+0000","last_clean":"2023-10-21T05:25:18.635654+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:25:18.635654+0000","last_undegraded":"2023-10-21T05:25:18.635654+0000","last_fullsized":"2023-10-21T05:25:18.635654+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8535,"log_dups_size":0,"ondisk_log_size":8535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1745","reported_seq":2875,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:15.888305+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:25:15.888305+0000","last_peered":"2023-10-21T05:25:15.888305+0000","last_clean":"2023-10-21T05:25:15.888305+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:25:15.888305+0000","last_undegraded":"2023-10-21T05:25:15.888305+0000","last_fullsized":"2023-10-21T05:25:15.888305+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1745,"log_dups_size":0,"ondisk_log_size":1745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1759","reported_seq":3011,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:18.633790+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:25:18.633790+0000","last_peered":"2023-10-21T05:25:18.633790+0000","last_clean":"2023-10-21T05:25:18.633790+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:25:18.633790+0000","last_undegraded":"2023-10-21T05:25:18.633790+0000","last_fullsized":"2023-10-21T05:25:18.633790+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1759,"log_dups_size":0,"ondisk_log_size":1759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'262967","reported_seq":527869,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:19.920986+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:25:19.920986+0000","last_peered":"2023-10-21T05:25:19.920986+0000","last_clean":"2023-10-21T05:25:19.920986+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:25:19.920986+0000","last_undegraded":"2023-10-21T05:25:19.920986+0000","last_fullsized":"2023-10-21T05:25:19.920986+0000","mapping_epoch":17,"log_start":"54'252915","ondisk_log_start":"54'252915","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523927040,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":264584,"num_read_kb":1031028,"num_write":262967,"num_write_kb":1051868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1829","reported_seq":2967,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:18.470146+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:25:18.470146+0000","last_peered":"2023-10-21T05:25:18.470146+0000","last_clean":"2023-10-21T05:25:18.470146+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:25:18.470146+0000","last_undegraded":"2023-10-21T05:25:18.470146+0000","last_fullsized":"2023-10-21T05:25:18.470146+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1829,"log_dups_size":0,"ondisk_log_size":1829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'245795","reported_seq":494200,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:18.980474+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:25:18.980474+0000","last_peered":"2023-10-21T05:25:18.980474+0000","last_clean":"2023-10-21T05:25:18.980474+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:25:18.980474+0000","last_undegraded":"2023-10-21T05:25:18.980474+0000","last_fullsized":"2023-10-21T05:25:18.980474+0000","mapping_epoch":17,"log_start":"54'235764","ondisk_log_start":"54'235764","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5191544832,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":248099,"num_read_kb":966432,"num_write":245795,"num_write_kb":983180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1140,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:13.513783+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:25:13.513783+0000","last_peered":"2023-10-21T05:25:13.513783+0000","last_clean":"2023-10-21T05:25:13.513783+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:25:13.513783+0000","last_undegraded":"2023-10-21T05:25:13.513783+0000","last_fullsized":"2023-10-21T05:25:13.513783+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'255609","reported_seq":513506,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:19.946493+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:25:19.946493+0000","last_peered":"2023-10-21T05:25:19.946493+0000","last_clean":"2023-10-21T05:25:19.946493+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:25:19.946493+0000","last_undegraded":"2023-10-21T05:25:19.946493+0000","last_fullsized":"2023-10-21T05:25:19.946493+0000","mapping_epoch":17,"log_start":"54'245517","ondisk_log_start":"54'245517","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5387075584,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":257590,"num_read_kb":1004216,"num_write":255609,"num_write_kb":1022436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4233,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:18.643955+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:25:18.643955+0000","last_peered":"2023-10-21T05:25:18.643955+0000","last_clean":"2023-10-21T05:25:18.643955+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:25:18.643955+0000","last_undegraded":"2023-10-21T05:25:18.643955+0000","last_fullsized":"2023-10-21T05:25:18.643955+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'249310","reported_seq":502183,"reported_epoch":54,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:25:18.963100+0000","last_change":"2023-10-21T05:17:44.157665+0000","last_active":"2023-10-21T05:25:18.963100+0000","last_peered":"2023-10-21T05:25:18.963100+0000","last_clean":"2023-10-21T05:25:18.963100+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:25:18.963100+0000","last_undegraded":"2023-10-21T05:25:18.963100+0000","last_fullsized":"2023-10-21T05:25:18.963100+0000","mapping_epoch":17,"log_start":"54'239292","ondisk_log_start":"54'239292","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:23.954438+0000","last_clean_scrub_stamp":"2023-10-21T04:05:23.954438+0000","objects_scrubbed":1228,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 454s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5262069779,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":252549,"num_read_kb":984328,"num_write":249310,"num_write_kb":997234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6785","reported_seq":10910,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:18.639750+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:25:18.639750+0000","last_peered":"2023-10-21T05:25:18.639750+0000","last_clean":"2023-10-21T05:25:18.639750+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:25:18.639750+0000","last_undegraded":"2023-10-21T05:25:18.639750+0000","last_fullsized":"2023-10-21T05:25:18.639750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6785,"log_dups_size":0,"ondisk_log_size":6785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":230,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:06.577736+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:22:06.577736+0000","last_peered":"2023-10-21T05:22:06.577736+0000","last_clean":"2023-10-21T05:22:06.577736+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:22:06.577736+0000","last_undegraded":"2023-10-21T05:22:06.577736+0000","last_fullsized":"2023-10-21T05:22:06.577736+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21364617235,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":1022822,"num_read_kb":3986004,"num_write":1013681,"num_write_kb":4054718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32061423616,"data_stored":32061423616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40193,"ondisk_log_size":40193,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":25894,"ondisk_log_size":25894,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739329,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11092364,"kb_used_data":10437120,"kb_used_omap":9,"kb_used_meta":655222,"kb_avail":83279476,"statfs":{"total":96636764160,"available":85278183424,"internally_reserved":0,"allocated":10687610880,"data_stored":10687306349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10227,"internal_metadata":670947341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.83499999999999996,"15min":0.84099999999999997},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.72899999999999998,"5min":8.0340000000000007,"15min":11.904999999999999},"last":2.5680000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.69599999999999995,"15min":0.71099999999999997},"min":{"1min":0.22500000000000001,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":10.096,"5min":10.096,"15min":12.252000000000001},"last":0.34300000000000003}]},{"osd":1,"last update":"Sat Oct 21 05:24:44 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.72899999999999998,"15min":0.71199999999999997},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.189},"max":{"1min":1.9419999999999999,"5min":6.5199999999999996,"15min":19.574000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.752,"15min":0.88},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19700000000000001},"max":{"1min":1.2529999999999999,"5min":6.8440000000000003,"15min":21.234999999999999},"last":3.29}]}]},{"osd":1,"up_from":8,"seq":34359739327,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12073924,"kb_used_data":10437820,"kb_used_omap":15,"kb_used_meta":1636080,"kb_avail":82297916,"statfs":{"total":96636764160,"available":84273065984,"internally_reserved":0,"allocated":10688327680,"data_stored":10687909884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1675346310},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,4,10,7,6,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:42 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":0.96399999999999997,"15min":0.876},"min":{"1min":0.192,"5min":0.155,"15min":0.13300000000000001},"max":{"1min":7.9130000000000003,"5min":10.311999999999999,"15min":14.682},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.628,"15min":0.56200000000000006},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.9249999999999998,"5min":9.1820000000000004,"15min":9.1820000000000004},"last":0.20999999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:24:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.71099999999999997,"15min":0.67300000000000004},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":0.44400000000000001,"5min":14.138,"15min":14.138},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.77300000000000002,"15min":0.69399999999999995},"min":{"1min":0.34300000000000003,"5min":0.28799999999999998,"15min":0.22600000000000001},"max":{"1min":0.53700000000000003,"5min":14.173,"15min":14.173},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359739329,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12061364,"kb_used_data":10437740,"kb_used_omap":12,"kb_used_meta":1623603,"kb_avail":82310476,"statfs":{"total":96636764160,"available":84285927424,"internally_reserved":0,"allocated":10688245760,"data_stored":10687880763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1662569897},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,0,0,6,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.51200000000000001,"15min":0.55800000000000005},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":0.76200000000000001,"5min":6.5449999999999999,"15min":6.5759999999999996},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.56200000000000006,"15min":0.629},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.52700000000000002,"5min":5.4059999999999997,"15min":6.6429999999999998},"last":0.37}]},{"osd":2,"last update":"Sat Oct 21 05:24:41 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.0189999999999999,"15min":0.93899999999999995},"min":{"1min":0.27600000000000002,"5min":0.23999999999999999,"15min":0.23499999999999999},"max":{"1min":20.888999999999999,"5min":20.888999999999999,"15min":20.888999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.728,"5min":0.96699999999999997,"15min":0.93899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.17599999999999999},"max":{"1min":22.489999999999998,"5min":22.489999999999998,"15min":22.489999999999998},"last":0.441}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10687172608,"data_stored":10687172608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10687131648,"data_stored":10687131648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10687119360,"data_stored":10687119360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:25:23.563 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:23.954438+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=11, tm_sec=17, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:25:23.563 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:25:43.564 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:25:44.124 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-21T05:25:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-21T05:25:44.167 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":2469,"stamp":"2023-10-21T05:25:43.852817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21365380963,"num_objects":5186,"num_object_clones":0,"num_object_copies":15493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":1032323,"num_read_kb":4005767,"num_write":1018092,"num_write_kb":4072859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66300,"ondisk_log_size":66300,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":35301088,"kb_used_data":31313092,"kb_used_omap":38,"kb_used_meta":3987929,"kb_avail":247814432,"statfs":{"total":289910292480,"available":253761978368,"internally_reserved":0,"allocated":32064606208,"data_stored":32063518884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39108,"internal_metadata":4083640124},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,7,5,14,11,11,3,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":57344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1866,"num_read_kb":7412,"num_write":2026,"num_write_kb":8104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002036"},"pg_stats":[{"pgid":"2.7","version":"54'5163","reported_seq":8320,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:39.113608+0000","last_change":"2023-10-21T05:11:30.757879+0000","last_active":"2023-10-21T05:25:39.113608+0000","last_peered":"2023-10-21T05:25:39.113608+0000","last_clean":"2023-10-21T05:25:39.113608+0000","last_became_active":"2023-10-21T04:05:16.931539+0000","last_became_peered":"2023-10-21T04:05:16.931539+0000","last_unstale":"2023-10-21T05:25:39.113608+0000","last_undegraded":"2023-10-21T05:25:39.113608+0000","last_fullsized":"2023-10-21T05:25:39.113608+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'4333","last_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_deep_scrub":"54'4333","last_deep_scrub_stamp":"2023-10-21T05:11:30.757677+0000","last_clean_scrub_stamp":"2023-10-21T05:11:30.757677+0000","objects_scrubbed":6,"log_size":5163,"log_dups_size":0,"ondisk_log_size":5163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:16:07.047061+0000","scrub_duration":0.033208753000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":746,"num_read_kb":629,"num_write":24,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'8555","reported_seq":13967,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:39.114206+0000","last_change":"2023-10-21T05:11:25.703449+0000","last_active":"2023-10-21T05:25:39.114206+0000","last_peered":"2023-10-21T05:25:39.114206+0000","last_clean":"2023-10-21T05:25:39.114206+0000","last_became_active":"2023-10-21T04:05:16.932479+0000","last_became_peered":"2023-10-21T04:05:16.932479+0000","last_unstale":"2023-10-21T05:25:39.114206+0000","last_undegraded":"2023-10-21T05:25:39.114206+0000","last_fullsized":"2023-10-21T05:25:39.114206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'7186","last_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_deep_scrub":"54'7186","last_deep_scrub_stamp":"2023-10-21T05:11:25.703266+0000","last_clean_scrub_stamp":"2023-10-21T05:11:25.703266+0000","objects_scrubbed":10,"log_size":8555,"log_dups_size":0,"ondisk_log_size":8555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:07:39.151884+0000","scrub_duration":0.085380684999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1657,"num_read_kb":1479,"num_write":82,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2914,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"54'1749","reported_seq":2884,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:33.997447+0000","last_change":"2023-10-21T05:11:23.644860+0000","last_active":"2023-10-21T05:25:33.997447+0000","last_peered":"2023-10-21T05:25:33.997447+0000","last_clean":"2023-10-21T05:25:33.997447+0000","last_became_active":"2023-10-21T04:05:16.931569+0000","last_became_peered":"2023-10-21T04:05:16.931569+0000","last_unstale":"2023-10-21T05:25:33.997447+0000","last_undegraded":"2023-10-21T05:25:33.997447+0000","last_fullsized":"2023-10-21T05:25:33.997447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1467","last_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_deep_scrub":"54'1467","last_deep_scrub_stamp":"2023-10-21T05:11:23.644684+0000","last_clean_scrub_stamp":"2023-10-21T05:11:23.644684+0000","objects_scrubbed":5,"log_size":1749,"log_dups_size":0,"ondisk_log_size":1749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:41:38.395247+0000","scrub_duration":0.028505671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":219,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":300,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"54'1763","reported_seq":3019,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:33.997862+0000","last_change":"2023-10-21T05:11:32.687331+0000","last_active":"2023-10-21T05:25:33.997862+0000","last_peered":"2023-10-21T05:25:33.997862+0000","last_clean":"2023-10-21T05:25:33.997862+0000","last_became_active":"2023-10-21T04:05:16.932716+0000","last_became_peered":"2023-10-21T04:05:16.932716+0000","last_unstale":"2023-10-21T05:25:33.997862+0000","last_undegraded":"2023-10-21T05:25:33.997862+0000","last_fullsized":"2023-10-21T05:25:33.997862+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1487","last_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_deep_scrub":"54'1487","last_deep_scrub_stamp":"2023-10-21T05:11:32.687191+0000","last_clean_scrub_stamp":"2023-10-21T05:11:32.687191+0000","objects_scrubbed":12,"log_size":1763,"log_dups_size":0,"ondisk_log_size":1763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:42.376086+0000","scrub_duration":0.034160439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":453,"num_read_kb":371,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":829,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"54'264044","reported_seq":529911,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:40.112839+0000","last_change":"2023-10-21T05:19:24.293406+0000","last_active":"2023-10-21T05:25:40.112839+0000","last_peered":"2023-10-21T05:25:40.112839+0000","last_clean":"2023-10-21T05:25:40.112839+0000","last_became_active":"2023-10-21T04:05:24.984561+0000","last_became_peered":"2023-10-21T04:05:24.984561+0000","last_unstale":"2023-10-21T05:25:40.112839+0000","last_undegraded":"2023-10-21T05:25:40.112839+0000","last_fullsized":"2023-10-21T05:25:40.112839+0000","mapping_epoch":17,"log_start":"54'253946","ondisk_log_start":"54'253946","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'245407","last_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_deep_scrub":"54'245407","last_deep_scrub_stamp":"2023-10-21T05:19:24.293219+0000","last_clean_scrub_stamp":"2023-10-21T05:19:24.293219+0000","objects_scrubbed":1324,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":481,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:49:22.077525+0000","scrub_duration":480.21525677400001,"objects_trimmed":0,"snaptrim_duration":0.0011252479999999999,"stat_sum":{"num_bytes":5523968000,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":265546,"num_read_kb":1034848,"num_write":264044,"num_write_kb":1056176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"54'1840","reported_seq":2982,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:38.738700+0000","last_change":"2023-10-21T05:11:21.294987+0000","last_active":"2023-10-21T05:25:38.738700+0000","last_peered":"2023-10-21T05:25:38.738700+0000","last_clean":"2023-10-21T05:25:38.738700+0000","last_became_active":"2023-10-21T04:05:16.933889+0000","last_became_peered":"2023-10-21T04:05:16.933889+0000","last_unstale":"2023-10-21T05:25:38.738700+0000","last_undegraded":"2023-10-21T05:25:38.738700+0000","last_fullsized":"2023-10-21T05:25:38.738700+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'1556","last_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_deep_scrub":"54'1556","last_deep_scrub_stamp":"2023-10-21T05:11:21.294193+0000","last_clean_scrub_stamp":"2023-10-21T05:11:21.294193+0000","objects_scrubbed":8,"log_size":1840,"log_dups_size":0,"ondisk_log_size":1840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:49.158388+0000","scrub_duration":0.10782891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":231,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1660,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"54'246780","reported_seq":496070,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:39.290261+0000","last_change":"2023-10-21T05:17:42.359624+0000","last_active":"2023-10-21T05:25:39.290261+0000","last_peered":"2023-10-21T05:25:39.290261+0000","last_clean":"2023-10-21T05:25:39.290261+0000","last_became_active":"2023-10-21T04:05:24.981707+0000","last_became_peered":"2023-10-21T04:05:24.981707+0000","last_unstale":"2023-10-21T05:25:39.290261+0000","last_undegraded":"2023-10-21T05:25:39.290261+0000","last_fullsized":"2023-10-21T05:25:39.290261+0000","mapping_epoch":17,"log_start":"54'236771","ondisk_log_start":"54'236771","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'224535","last_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_deep_scrub":"54'224535","last_deep_scrub_stamp":"2023-10-21T05:17:42.359409+0000","last_clean_scrub_stamp":"2023-10-21T05:17:42.359409+0000","objects_scrubbed":1244,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":368,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:54:44.040946+0000","scrub_duration":367.54595820700001,"objects_trimmed":0,"snaptrim_duration":0.0010168530000000001,"stat_sum":{"num_bytes":5191561216,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":248983,"num_read_kb":969944,"num_write":246780,"num_write_kb":987120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'24","reported_seq":1143,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:28.874454+0000","last_change":"2023-10-21T05:11:26.996508+0000","last_active":"2023-10-21T05:25:28.874454+0000","last_peered":"2023-10-21T05:25:28.874454+0000","last_clean":"2023-10-21T05:25:28.874454+0000","last_became_active":"2023-10-21T04:05:16.933471+0000","last_became_peered":"2023-10-21T04:05:16.933471+0000","last_unstale":"2023-10-21T05:25:28.874454+0000","last_undegraded":"2023-10-21T05:25:28.874454+0000","last_fullsized":"2023-10-21T05:25:28.874454+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'24","last_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_deep_scrub":"54'24","last_deep_scrub_stamp":"2023-10-21T05:11:26.996337+0000","last_clean_scrub_stamp":"2023-10-21T05:11:26.996337+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:31:49.930498+0000","scrub_duration":0.093603776999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":211,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'256561","reported_seq":515393,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:40.112308+0000","last_change":"2023-10-21T05:17:59.949673+0000","last_active":"2023-10-21T05:25:40.112308+0000","last_peered":"2023-10-21T05:25:40.112308+0000","last_clean":"2023-10-21T05:25:40.112308+0000","last_became_active":"2023-10-21T04:05:24.984525+0000","last_became_peered":"2023-10-21T04:05:24.984525+0000","last_unstale":"2023-10-21T05:25:40.112308+0000","last_undegraded":"2023-10-21T05:25:40.112308+0000","last_fullsized":"2023-10-21T05:25:40.112308+0000","mapping_epoch":17,"log_start":"54'246526","ondisk_log_start":"54'246526","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'234259","last_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_deep_scrub":"54'234259","last_deep_scrub_stamp":"2023-10-21T05:17:59.949296+0000","last_clean_scrub_stamp":"2023-10-21T05:17:59.949296+0000","objects_scrubbed":1291,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":395,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:51:25.205771+0000","scrub_duration":394.85708301099999,"objects_trimmed":0,"snaptrim_duration":0.0009956190000000001,"stat_sum":{"num_bytes":5387149312,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":258527,"num_read_kb":1007948,"num_write":256561,"num_write_kb":1026244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"54'66","reported_seq":4245,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:33.997767+0000","last_change":"2023-10-21T05:11:27.933326+0000","last_active":"2023-10-21T05:25:33.997767+0000","last_peered":"2023-10-21T05:25:33.997767+0000","last_clean":"2023-10-21T05:25:33.997767+0000","last_became_active":"2023-10-21T04:05:16.932458+0000","last_became_peered":"2023-10-21T04:05:16.932458+0000","last_unstale":"2023-10-21T05:25:33.997767+0000","last_undegraded":"2023-10-21T05:25:33.997767+0000","last_fullsized":"2023-10-21T05:25:33.997767+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'66","last_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_deep_scrub":"54'66","last_deep_scrub_stamp":"2023-10-21T05:11:27.933146+0000","last_clean_scrub_stamp":"2023-10-21T05:11:27.933146+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:19.951857+0000","scrub_duration":0.034708112999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":999,"num_read_kb":840,"num_write":31,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1203,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"54'250307","reported_seq":504140,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:39.294738+0000","last_change":"2023-10-21T05:25:26.505562+0000","last_active":"2023-10-21T05:25:39.294738+0000","last_peered":"2023-10-21T05:25:39.294738+0000","last_clean":"2023-10-21T05:25:39.294738+0000","last_became_active":"2023-10-21T04:05:24.984739+0000","last_became_peered":"2023-10-21T04:05:24.984739+0000","last_unstale":"2023-10-21T05:25:39.294738+0000","last_undegraded":"2023-10-21T05:25:39.294738+0000","last_fullsized":"2023-10-21T05:25:39.294738+0000","mapping_epoch":17,"log_start":"54'240209","ondisk_log_start":"54'240209","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'249666","last_scrub_stamp":"2023-10-21T05:25:26.505382+0000","last_deep_scrub":"54'249666","last_deep_scrub_stamp":"2023-10-21T05:25:26.505382+0000","last_clean_scrub_stamp":"2023-10-21T05:25:26.505382+0000","objects_scrubbed":1262,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":463,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:45:53.403490+0000","scrub_duration":462.29621477900002,"objects_trimmed":0,"snaptrim_duration":0.002024484,"stat_sum":{"num_bytes":5262110739,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":253501,"num_read_kb":988120,"num_write":250307,"num_write_kb":1001222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"54'6801","reported_seq":10942,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:25:39.114928+0000","last_change":"2023-10-21T05:11:29.105972+0000","last_active":"2023-10-21T05:25:39.114928+0000","last_peered":"2023-10-21T05:25:39.114928+0000","last_clean":"2023-10-21T05:25:39.114928+0000","last_became_active":"2023-10-21T04:05:16.931528+0000","last_became_peered":"2023-10-21T04:05:16.931528+0000","last_unstale":"2023-10-21T05:25:39.114928+0000","last_undegraded":"2023-10-21T05:25:39.114928+0000","last_fullsized":"2023-10-21T05:25:39.114928+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'5725","last_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_deep_scrub":"54'5725","last_deep_scrub_stamp":"2023-10-21T05:11:29.105769+0000","last_clean_scrub_stamp":"2023-10-21T05:11:29.105769+0000","objects_scrubbed":10,"log_size":6801,"log_dups_size":0,"ondisk_log_size":6801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T17:03:09.109728+0000","scrub_duration":0.38750130500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":986,"num_read_kb":836,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1204,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"54'99","reported_seq":230,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-21T05:22:06.577736+0000","last_change":"2023-10-21T05:11:22.695206+0000","last_active":"2023-10-21T05:22:06.577736+0000","last_peered":"2023-10-21T05:22:06.577736+0000","last_clean":"2023-10-21T05:22:06.577736+0000","last_became_active":"2023-10-21T04:05:12.083812+0000","last_became_peered":"2023-10-21T04:05:12.083812+0000","last_unstale":"2023-10-21T05:22:06.577736+0000","last_undegraded":"2023-10-21T05:22:06.577736+0000","last_fullsized":"2023-10-21T05:22:06.577736+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'99","last_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_deep_scrub":"54'99","last_deep_scrub_stamp":"2023-10-21T05:11:22.694573+0000","last_clean_scrub_stamp":"2023-10-21T05:11:22.694573+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:28:50.882417+0000","scrub_duration":0.049253313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21364789267,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":1026557,"num_read_kb":4000860,"num_write":1017692,"num_write_kb":4070762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32061743104,"data_stored":32061743104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40240,"ondisk_log_size":40240,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":5660,"num_read_kb":4816,"num_write":263,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32647,"internal_metadata":0},"log_size":25961,"ondisk_log_size":25961,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739333,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11110544,"kb_used_data":10437252,"kb_used_omap":9,"kb_used_meta":673270,"kb_avail":83261296,"statfs":{"total":96636764160,"available":85259567104,"internally_reserved":0,"allocated":10687746048,"data_stored":10687441517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10227,"internal_metadata":689428493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.83499999999999996,"15min":0.84099999999999997},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.72899999999999998,"5min":8.0340000000000007,"15min":11.904999999999999},"last":0.501},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.69599999999999995,"15min":0.71099999999999997},"min":{"1min":0.22500000000000001,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":10.096,"5min":10.096,"15min":12.252000000000001},"last":0.27500000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:24:44 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.72899999999999998,"15min":0.71199999999999997},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.189},"max":{"1min":1.9419999999999999,"5min":6.5199999999999996,"15min":19.574000000000002},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.752,"15min":0.88},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19700000000000001},"max":{"1min":1.2529999999999999,"5min":6.8440000000000003,"15min":21.234999999999999},"last":0.83399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739331,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12092076,"kb_used_data":10437924,"kb_used_omap":15,"kb_used_meta":1654128,"kb_avail":82279764,"statfs":{"total":96636764160,"available":84254478336,"internally_reserved":0,"allocated":10688434176,"data_stored":10688016380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15994,"internal_metadata":1693827462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,5,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:42 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":0.96399999999999997,"15min":0.876},"min":{"1min":0.192,"5min":0.155,"15min":0.13300000000000001},"max":{"1min":7.9130000000000003,"5min":10.311999999999999,"15min":14.682},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.628,"15min":0.56200000000000006},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.9249999999999998,"5min":9.1820000000000004,"15min":9.1820000000000004},"last":0.252}]},{"osd":2,"last update":"Sat Oct 21 05:24:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.71099999999999997,"15min":0.67300000000000004},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":0.44400000000000001,"5min":14.138,"15min":14.138},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.77300000000000002,"15min":0.69399999999999995},"min":{"1min":0.34300000000000003,"5min":0.28799999999999998,"15min":0.22600000000000001},"max":{"1min":0.53700000000000003,"5min":14.173,"15min":14.173},"last":0.39700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739333,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12098468,"kb_used_data":10437916,"kb_used_omap":12,"kb_used_meta":1660531,"kb_avail":82273372,"statfs":{"total":96636764160,"available":84247932928,"internally_reserved":0,"allocated":10688425984,"data_stored":10688060987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12887,"internal_metadata":1700384169},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,8,6,6,3,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.51200000000000001,"15min":0.55800000000000005},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":0.76200000000000001,"5min":6.5449999999999999,"15min":6.5759999999999996},"last":0.315},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.56200000000000006,"15min":0.629},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.52700000000000002,"5min":5.4059999999999997,"15min":6.6429999999999998},"last":0.28499999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:24:41 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.0189999999999999,"15min":0.93899999999999995},"min":{"1min":0.27600000000000002,"5min":0.23999999999999999,"15min":0.23499999999999999},"max":{"1min":20.888999999999999,"5min":20.888999999999999,"15min":20.888999999999999},"last":2.8809999999999998},{"interface":"front","average":{"1min":1.728,"5min":0.96699999999999997,"15min":0.93899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.17599999999999999},"max":{"1min":22.489999999999998,"5min":22.489999999999998,"15min":22.489999999999998},"last":2.919}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11293,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14401,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6953,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10687250432,"data_stored":10687250432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10687238144,"data_stored":10687238144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10687254528,"data_stored":10687254528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:25:44.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-21T05:25:44.871 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-21T05:25:44.871 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-21T05:25:44.872 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-21T05:25:44.872 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:25:49.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-21T05:25:50.974 INFO:tasks.ceph.osd.0:Stopped 2023-10-21T05:25:50.975 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-21T05:25:50.976 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:25:51.073 INFO:tasks.ceph.osd.1:Stopped 2023-10-21T05:25:51.074 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-21T05:25:51.074 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:25:54.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-21T05:25:54.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-21T05:25:57.177 INFO:tasks.ceph.osd.2:Stopped 2023-10-21T05:25:57.178 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-21T05:25:57.178 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-21T05:25:57.179 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:25:57.210 INFO:tasks.ceph.mgr.x:Stopped 2023-10-21T05:25:57.213 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-21T05:25:57.214 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-21T05:25:57.214 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:25:57.270 INFO:tasks.ceph.mon.a:Stopped 2023-10-21T05:25:57.271 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-21T05:25:57.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-21T05:25:57.400 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-21T05:25:57.400 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-21T05:25:57.907 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-21T05:25:57.908 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-21T05:25:58.144 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-21T05:25:58.144 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-21T05:25:58.388 INFO:tasks.ceph:Archiving mon data... 2023-10-21T05:25:58.390 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433369/data/mon.a.tgz 2023-10-21T05:25:58.391 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2023-10-21T05:25:58.414 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.Y3vvYL9Q6B 2023-10-21T05:25:58.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Y3vvYL9Q6B 2023-10-21T05:25:58.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.Y3vvYL9Q6B 2023-10-21T05:25:58.645 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.Y3vvYL9Q6B is 337KB 2023-10-21T05:25:58.755 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.Y3vvYL9Q6B 2023-10-21T05:25:58.760 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-21T05:25:58.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-21T05:25:58.845 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-21T05:25:58.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-21T05:25:58.915 INFO:teuthology.orchestra.run.smithi172.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-21T05:25:58.918 INFO:tasks.ceph:Archiving crash dumps... 2023-10-21T05:25:58.921 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433369/remote/smithi172/crash 2023-10-21T05:25:58.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-21T05:25:58.969 INFO:tasks.ceph:Compressing logs... 2023-10-21T05:25:58.970 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-21T05:25:59.039 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48900.log 2023-10-21T05:25:59.045 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40358.log 2023-10-21T05:25:59.045 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40554.log 2023-10-21T05:25:59.046 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48900.log.gz 2023-10-21T05:25:59.046 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44459.log 2023-10-21T05:25:59.046 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42587.log 2023-10-21T05:25:59.047 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40554.log.gz 2023-10-21T05:25:59.047 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40358.log.gz 2023-10-21T05:25:59.047 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44459.log.gz 2023-10-21T05:25:59.047 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50634.log 2023-10-21T05:25:59.048 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45884.log 2023-10-21T05:25:59.048 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42587.log.gz 2023-10-21T05:25:59.048 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50634.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.50634.log.gz -- 2023-10-21T05:25:59.049 INFO:teuthology.orchestra.run.smithi172.stderr: /var/log/ceph/ceph-client.admin.48443.log 2023-10-21T05:25:59.049 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52429.log 2023-10-21T05:25:59.050 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45884.log.gz 2023-10-21T05:25:59.050 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47017.log 2023-10-21T05:25:59.051 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48443.log.gz 2023-10-21T05:25:59.051 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49112.log 2023-10-21T05:25:59.052 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52429.log.gz 2023-10-21T05:25:59.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39216.log 2023-10-21T05:25:59.053 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47017.log.gz 2023-10-21T05:25:59.053 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49112.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.49112.log.gz -- 2023-10-21T05:25:59.054 INFO:teuthology.orchestra.run.smithi172.stderr: /var/log/ceph/ceph-client.admin.52655.log 2023-10-21T05:25:59.054 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40466.log 2023-10-21T05:25:59.055 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39216.log.gz 2023-10-21T05:25:59.055 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52356.log 2023-10-21T05:25:59.056 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52655.log.gz 2023-10-21T05:25:59.056 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5/var/log/ceph/ceph-client.admin.40466.log: --verbose -- /var/log/ceph/ceph-client.admin.46434.log 2023-10-21T05:25:59.057 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40466.log.gz 2023-10-21T05:25:59.057 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49509.log 2023-10-21T05:25:59.058 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52356.log.gz 2023-10-21T05:25:59.058 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52035.log 2023-10-21T05:25:59.059 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46434.log.gz 2023-10-21T05:25:59.226 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48319.log 2023-10-21T05:25:59.227 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49509.log.gz 2023-10-21T05:25:59.227 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54103.log 2023-10-21T05:25:59.229 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52035.log.gz 2023-10-21T05:25:59.229 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44580.log 2023-10-21T05:25:59.229 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48319.log.gz 2023-10-21T05:25:59.230 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.54103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47713.log 2023-10-21T05:25:59.230 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54103.log.gz 2023-10-21T05:25:59.230 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40986.log 2023-10-21T05:25:59.231 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44580.log.gz 2023-10-21T05:25:59.231 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42207.log 2023-10-21T05:25:59.231 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47713.log.gz 2023-10-21T05:25:59.232 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43089.log 2023-10-21T05:25:59.232 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52839.log 2023-10-21T05:25:59.232 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42207.log.gz 2023-10-21T05:25:59.232 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40986.log.gz 2023-10-21T05:25:59.233 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49475.log 2023-10-21T05:25:59.233 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43089.log.gz 2023-10-21T05:25:59.233 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46192.log 2023-10-21T05:25:59.234 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52839.log.gz 2023-10-21T05:25:59.234 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39087.log 2023-10-21T05:25:59.234 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49475.log.gz 2023-10-21T05:25:59.234 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50694.log 2023-10-21T05:25:59.235 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46192.log.gz 2023-10-21T05:25:59.235 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43876.log 2023-10-21T05:25:59.235 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39087.log.gz 2023-10-21T05:25:59.235 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50694.log.gz 2023-10-21T05:25:59.236 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48079.log 2023-10-21T05:25:59.236 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5/var/log/ceph/ceph-client.admin.43876.log: --verbose -- /var/log/ceph/ceph-client.admin.51335.log 2023-10-21T05:25:59.236 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43876.log.gz 2023-10-21T05:25:59.237 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49692.log 2023-10-21T05:25:59.237 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48079.log.gz 2023-10-21T05:25:59.237 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51335.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.44940.log 2023-10-21T05:25:59.237 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51335.log.gz 2023-10-21T05:25:59.238 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45431.log 2023-10-21T05:25:59.238 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49692.log.gz 2023-10-21T05:25:59.238 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42098.log 2023-10-21T05:25:59.239 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44940.log.gz 2023-10-21T05:25:59.239 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45431.log.gz 2023-10-21T05:25:59.239 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51669.log 2023-10-21T05:25:59.239 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46099.log 2023-10-21T05:25:59.240 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42098.log.gz 2023-10-21T05:25:59.240 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41023.log 2023-10-21T05:25:59.240 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51669.log.gz 2023-10-21T05:25:59.240 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43301.log 2023-10-21T05:25:59.241 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46099.log.gz 2023-10-21T05:25:59.241 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41023.log: /var/log/ceph/ceph-client.admin.50817.log 2023-10-21T05:25:59.241 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41023.log.gz 2023-10-21T05:25:59.242 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53297.log 2023-10-21T05:25:59.242 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43301.log.gz 2023-10-21T05:25:59.242 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40753.log 2023-10-21T05:25:59.242 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50817.log.gz 2023-10-21T05:25:59.243 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48688.log 2023-10-21T05:25:59.243 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53297.log.gz 2023-10-21T05:25:59.243 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40898.log 2023-10-21T05:25:59.244 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40753.log.gz 2023-10-21T05:25:59.244 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48688.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30865.log 2023-10-21T05:25:59.244 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48688.log.gz 2023-10-21T05:25:59.244 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44730.log 2023-10-21T05:25:59.245 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40898.log.gz 2023-10-21T05:25:59.245 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.tmp-client.admin.30865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52551.log 2023-10-21T05:25:59.245 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30865.log.gz 2023-10-21T05:25:59.245 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41666.log 2023-10-21T05:25:59.246 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44730.log.gz 2023-10-21T05:25:59.246 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45975.log 2023-10-21T05:25:59.246 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52551.log.gz 2023-10-21T05:25:59.247 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47500.log 2023-10-21T05:25:59.247 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48139.log 2023-10-21T05:25:59.247 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45975.log.gz 2023-10-21T05:25:59.247 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48749.log 2023-10-21T05:25:59.248 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47500.log.gz 2023-10-21T05:25:59.248 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42999.log 2023-10-21T05:25:59.248 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48139.log.gz 2023-10-21T05:25:59.249 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48749.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-21T05:25:59.249 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48749.log.gz 2023-10-21T05:25:59.249 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42909.log 2023-10-21T05:25:59.249 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42999.log.gz 2023-10-21T05:25:59.250 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40173.log 2023-10-21T05:25:59.250 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50940.log 2023-10-21T05:25:59.250 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42909.log.gz 2023-10-21T05:25:59.251 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45061.log 2023-10-21T05:25:59.251 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40173.log.gz 2023-10-21T05:25:59.251 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40483.log 2023-10-21T05:25:59.251 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50940.log.gz 2023-10-21T05:25:59.252 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45061.log.gz 2023-10-21T05:25:59.252 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43726.log 2023-10-21T05:25:59.252 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39328.log 2023-10-21T05:25:59.253 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40483.log.gz 2023-10-21T05:25:59.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53323.log 2023-10-21T05:25:59.253 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43726.log.gz 2023-10-21T05:25:59.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44427.log 2023-10-21T05:25:59.254 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39328.log.gz 2023-10-21T05:25:59.254 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47773.log 2023-10-21T05:25:59.254 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53323.log.gz 2023-10-21T05:25:59.255 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46224.log 2023-10-21T05:25:59.255 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44427.log.gz 2023-10-21T05:25:59.255 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47773.log.gz 2023-10-21T05:25:59.255 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52210.log 2023-10-21T05:25:59.256 INFO:teuthology.orchestra.run.smithi172.stderr:gzip/var/log/ceph/ceph-client.admin.46224.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43909.log 2023-10-21T05:25:59.256 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46224.log.gz 2023-10-21T05:25:59.256 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50209.log 2023-10-21T05:25:59.257 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52210.log.gz 2023-10-21T05:25:59.257 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52228.log 2023-10-21T05:25:59.257 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43909.log.gz 2023-10-21T05:25:59.257 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45400.log 2023-10-21T05:25:59.258 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50209.log.gz 2023-10-21T05:25:59.258 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53456.log 2023-10-21T05:25:59.258 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52228.log.gz 2023-10-21T05:25:59.258 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45400.log.gz 2023-10-21T05:25:59.259 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44029.log 2023-10-21T05:25:59.259 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40878.log 2023-10-21T05:25:59.259 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53456.log.gz 2023-10-21T05:25:59.260 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48289.log 2023-10-21T05:25:59.260 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44029.log.gz 2023-10-21T05:25:59.260 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44336.log 2023-10-21T05:25:59.260 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40878.log.gz 2023-10-21T05:25:59.261 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48289.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.44273.log 2023-10-21T05:25:59.261 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48289.log.gz 2023-10-21T05:25:59.261 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52003.log 2023-10-21T05:25:59.262 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44336.log.gz 2023-10-21T05:25:59.262 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41666.log.gz 2023-10-21T05:25:59.262 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51213.log 2023-10-21T05:25:59.262 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44273.log.gz 2023-10-21T05:25:59.263 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53912.log 2023-10-21T05:25:59.263 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52003.log.gz 2023-10-21T05:25:59.263 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45370.log 2023-10-21T05:25:59.264 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51213.log.gz 2023-10-21T05:25:59.264 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.53912.log: -- /var/log/ceph/ceph-client.admin.52099.log 2023-10-21T05:25:59.264 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53912.log.gz 2023-10-21T05:25:59.264 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46068.log 2023-10-21T05:25:59.265 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45370.log.gz 2023-10-21T05:25:59.265 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50179.log 2023-10-21T05:25:59.265 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52099.log.gz 2023-10-21T05:25:59.265 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45280.log 2023-10-21T05:25:59.266 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46068.log.gz 2023-10-21T05:25:59.266 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48563.log 2023-10-21T05:25:59.266 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50179.log.gz 2023-10-21T05:25:59.267 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45280.log.gz 2023-10-21T05:25:59.267 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53245.log 2023-10-21T05:25:59.267 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50849.log 2023-10-21T05:25:59.267 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48563.log.gz 2023-10-21T05:25:59.268 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49903.log 2023-10-21T05:25:59.268 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53245.log.gz 2023-10-21T05:25:59.268 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49872.log 2023-10-21T05:25:59.269 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50849.log.gz 2023-10-21T05:25:59.269 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40520.log 2023-10-21T05:25:59.269 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49903.log.gz 2023-10-21T05:25:59.269 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43606.log 2023-10-21T05:25:59.270 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49872.log.gz 2023-10-21T05:25:59.270 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52318.log 2023-10-21T05:25:59.270 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40520.log.gz 2023-10-21T05:25:59.271 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50454.log 2023-10-21T05:25:59.271 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43606.log.gz 2023-10-21T05:25:59.271 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40193.log 2023-10-21T05:25:59.271 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52318.log.gz 2023-10-21T05:25:59.272 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41148.log 2023-10-21T05:25:59.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50454.log.gz 2023-10-21T05:25:59.272 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47259.log 2023-10-21T05:25:59.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40193.log.gz 2023-10-21T05:25:59.273 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41148.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.41148.log.gz 2023-10-21T05:25:59.273 INFO:teuthology.orchestra.run.smithi172.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45248.log 2023-10-21T05:25:59.273 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48779.log 2023-10-21T05:25:59.274 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47259.log.gz 2023-10-21T05:25:59.274 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5/var/log/ceph/ceph-client.admin.45248.log: --verbose -- /var/log/ceph/ceph-client.admin.47229.log 2023-10-21T05:25:59.274 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45248.log.gz 2023-10-21T05:25:59.274 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52502.log 2023-10-21T05:25:59.275 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48779.log.gz 2023-10-21T05:25:59.275 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43119.log 2023-10-21T05:25:59.275 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47229.log.gz 2023-10-21T05:25:59.275 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52502.log.gz 2023-10-21T05:25:59.276 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50664.log 2023-10-21T05:25:59.276 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50574.log 2023-10-21T05:25:59.276 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43119.log.gz 2023-10-21T05:25:59.277 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44366.log 2023-10-21T05:25:59.277 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50664.log.gz 2023-10-21T05:25:59.277 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47409.log 2023-10-21T05:25:59.277 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50574.log.gz 2023-10-21T05:25:59.278 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53401.log 2023-10-21T05:25:59.278 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44366.log.gz 2023-10-21T05:25:59.278 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47409.log.gz 2023-10-21T05:25:59.279 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34550.log 2023-10-21T05:25:59.279 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42150.log 2023-10-21T05:25:59.279 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53401.log.gz 2023-10-21T05:25:59.279 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47319.log 2023-10-21T05:25:59.280 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34550.log.gz 2023-10-21T05:25:59.280 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42150.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-21T05:25:59.280 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42150.log.gz 2023-10-21T05:25:59.281 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43029.log 2023-10-21T05:25:59.281 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47319.log.gz 2023-10-21T05:25:59.281 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43666.log 2023-10-21T05:25:59.281 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46829.log 2023-10-21T05:25:59.282 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43029.log.gz 2023-10-21T05:25:59.282 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43666.log.gz 2023-10-21T05:25:59.282 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39553.log 2023-10-21T05:25:59.283 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2023-10-21T05:25:59.283 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46829.log.gz 2023-10-21T05:25:59.283 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47621.log 2023-10-21T05:25:59.283 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39553.log.gz 2023-10-21T05:25:59.284 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 2023-10-21T05:25:59.284 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49353.log 2023-10-21T05:25:59.284 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49445.log 2023-10-21T05:25:59.285 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47621.log.gz 2023-10-21T05:25:59.285 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51060.log 2023-10-21T05:25:59.285 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49353.log.gz 2023-10-21T05:25:59.285 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49445.log.gz 2023-10-21T05:25:59.286 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42341.log 2023-10-21T05:25:59.286 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48018.log 2023-10-21T05:25:59.286 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51060.log.gz 2023-10-21T05:25:59.286 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45914.log 2023-10-21T05:25:59.287 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42341.log.gz 2023-10-21T05:25:59.287 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53966.log 2023-10-21T05:25:59.287 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48018.log.gz 2023-10-21T05:25:59.288 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53616.log 2023-10-21T05:25:59.288 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45914.log.gz 2023-10-21T05:25:59.288 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51702.log 2023-10-21T05:25:59.288 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53966.log.gz 2023-10-21T05:25:59.289 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45945.log 2023-10-21T05:25:59.289 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53616.log.gz 2023-10-21T05:25:59.289 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51702.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-21T05:25:59.290 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51702.log.gz 2023-10-21T05:25:59.290 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44396.log 2023-10-21T05:25:59.290 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45945.log.gz 2023-10-21T05:25:59.290 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54046.log 2023-10-21T05:25:59.291 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39502.log 2023-10-21T05:25:59.291 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44396.log.gz 2023-10-21T05:25:59.291 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.54046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45672.log 2023-10-21T05:25:59.292 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54046.log.gz 2023-10-21T05:25:59.292 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48169.log 2023-10-21T05:25:59.292 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39502.log.gz 2023-10-21T05:25:59.292 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34574.log 2023-10-21T05:25:59.293 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45672.log.gz 2023-10-21T05:25:59.293 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39528.log 2023-10-21T05:25:59.293 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48169.log.gz 2023-10-21T05:25:59.293 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47349.log 2023-10-21T05:25:59.294 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34574.log.gz 2023-10-21T05:25:59.294 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46860.log 2023-10-21T05:25:59.294 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39528.log.gz 2023-10-21T05:25:59.295 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49021.log 2023-10-21T05:25:59.295 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47349.log.gz 2023-10-21T05:25:59.295 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40628.log 2023-10-21T05:25:59.296 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46860.log.gz 2023-10-21T05:25:59.296 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48473.log 2023-10-21T05:25:59.296 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49021.log.gz 2023-10-21T05:25:59.296 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40628.log.gz 2023-10-21T05:25:59.297 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50270.log 2023-10-21T05:25:59.297 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48473.log.gz 2023-10-21T05:25:59.297 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50363.log 2023-10-21T05:25:59.298 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48961.log 2023-10-21T05:25:59.298 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50270.log.gz 2023-10-21T05:25:59.298 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47958.log 2023-10-21T05:25:59.299 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50363.log.gz 2023-10-21T05:25:59.299 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43059.log 2023-10-21T05:25:59.299 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48961.log.gz 2023-10-21T05:25:59.299 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47958.log.gzgzip 2023-10-21T05:25:59.300 INFO:teuthology.orchestra.run.smithi172.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52246.log 2023-10-21T05:25:59.300 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43969.log 2023-10-21T05:25:59.300 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43059.log.gz 2023-10-21T05:25:59.301 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42510.log 2023-10-21T05:25:59.301 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52246.log.gz 2023-10-21T05:25:59.301 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46614.log 2023-10-21T05:25:59.301 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43969.log.gz 2023-10-21T05:25:59.302 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54179.log 2023-10-21T05:25:59.302 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42510.log.gz 2023-10-21T05:25:59.302 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50056.log 2023-10-21T05:25:59.302 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46614.log.gz 2023-10-21T05:25:59.303 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.54179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39629.log 2023-10-21T05:25:59.303 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54179.log.gz 2023-10-21T05:25:59.303 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42969.log 2023-10-21T05:25:59.304 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50056.log.gz 2023-10-21T05:25:59.304 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40574.log 2023-10-21T05:25:59.304 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39629.log.gz 2023-10-21T05:25:59.304 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45001.log 2023-10-21T05:25:59.305 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42969.log.gz 2023-10-21T05:25:59.305 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40122.log 2023-10-21T05:25:59.305 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40574.log.gz 2023-10-21T05:25:59.305 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45793.log 2023-10-21T05:25:59.306 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45001.log.gz 2023-10-21T05:25:59.306 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42540.log 2023-10-21T05:25:59.306 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40122.log.gz 2023-10-21T05:25:59.307 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42639.log 2023-10-21T05:25:59.307 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45793.log.gz 2023-10-21T05:25:59.307 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50149.log 2023-10-21T05:25:59.307 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42540.log.gz 2023-10-21T05:25:59.308 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44971.log 2023-10-21T05:25:59.308 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42639.log.gz 2023-10-21T05:25:59.308 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49601.log 2023-10-21T05:25:59.309 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50149.log.gz 2023-10-21T05:25:59.309 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42450.log 2023-10-21T05:25:59.309 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44971.log.gz 2023-10-21T05:25:59.310 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40844.log 2023-10-21T05:25:59.310 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49601.log.gz 2023-10-21T05:25:59.310 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41168.log 2023-10-21T05:25:59.310 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42450.log.gz 2023-10-21T05:25:59.311 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40500.log 2023-10-21T05:25:59.311 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40844.log.gz 2023-10-21T05:25:59.311 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46796.log 2023-10-21T05:25:59.312 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41168.log.gz 2023-10-21T05:25:59.312 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5/var/log/ceph/ceph-client.admin.40500.log: --verbose -- /var/log/ceph/ceph-client.admin.44700.log 2023-10-21T05:25:59.312 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40500.log.gz 2023-10-21T05:25:59.312 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46796.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.46796.log.gz 2023-10-21T05:25:59.313 INFO:teuthology.orchestra.run.smithi172.stderr: /var/log/ceph/ceph-client.admin.53271.log 2023-10-21T05:25:59.313 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46982.log 2023-10-21T05:25:59.313 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44700.log.gz 2023-10-21T05:25:59.314 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48349.log 2023-10-21T05:25:59.314 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53271.log.gz 2023-10-21T05:25:59.314 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43149.log 2023-10-21T05:25:59.315 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46982.log.gz 2023-10-21T05:25:59.315 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42696.log 2023-10-21T05:25:59.315 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48349.log.gz 2023-10-21T05:25:59.316 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49414.log 2023-10-21T05:25:59.316 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43149.log.gz 2023-10-21T05:25:59.316 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34456.log 2023-10-21T05:25:59.316 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42696.log.gz 2023-10-21T05:25:59.317 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42297.log 2023-10-21T05:25:59.317 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49414.log.gz 2023-10-21T05:25:59.317 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43271.log 2023-10-21T05:25:59.317 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40537.log 2023-10-21T05:25:59.318 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42297.log.gz 2023-10-21T05:25:59.318 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53028.log 2023-10-21T05:25:59.318 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43271.log.gz 2023-10-21T05:25:59.319 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51244.log 2023-10-21T05:25:59.319 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40537.log.gz 2023-10-21T05:25:59.319 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50604.log 2023-10-21T05:25:59.319 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53028.log.gz 2023-10-21T05:25:59.320 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51244.log.gz 2023-10-21T05:25:59.320 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52173.log 2023-10-21T05:25:59.320 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50604.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.52448.log 2023-10-21T05:25:59.321 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50604.log.gz 2023-10-21T05:25:59.321 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46344.log 2023-10-21T05:25:59.321 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52173.log.gz 2023-10-21T05:25:59.321 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47804.log 2023-10-21T05:25:59.322 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52448.log.gz 2023-10-21T05:25:59.322 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48259.log 2023-10-21T05:25:59.322 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46344.log.gz 2023-10-21T05:25:59.322 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45183.log 2023-10-21T05:25:59.323 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47804.log.gz 2023-10-21T05:25:59.323 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49051.log 2023-10-21T05:25:59.323 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48259.log.gz 2023-10-21T05:25:59.324 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52192.log 2023-10-21T05:25:59.324 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45183.log.gz 2023-10-21T05:25:59.324 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34456.log.gz 2023-10-21T05:25:59.324 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49051.log.gz 2023-10-21T05:25:59.325 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46314.log 2023-10-21T05:25:59.325 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46584.log 2023-10-21T05:25:59.325 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52192.log.gz 2023-10-21T05:25:59.326 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53939.log 2023-10-21T05:25:59.326 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46314.log.gz 2023-10-21T05:25:59.326 INFO:teuthology.orchestra.run.smithi172.stderr:gzip/var/log/ceph/ceph-client.admin.46584.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39190.log 2023-10-21T05:25:59.326 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46584.log.gz 2023-10-21T05:25:59.327 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47682.log 2023-10-21T05:25:59.327 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53939.log.gz 2023-10-21T05:25:59.327 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51732.log 2023-10-21T05:25:59.328 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39190.log.gz 2023-10-21T05:25:59.328 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43392.log 2023-10-21T05:25:59.328 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47682.log.gz 2023-10-21T05:25:59.328 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44549.log 2023-10-21T05:25:59.329 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51732.log.gz 2023-10-21T05:25:59.329 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52759.log 2023-10-21T05:25:59.329 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43392.log.gz 2023-10-21T05:25:59.330 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42393.log 2023-10-21T05:25:59.330 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44549.log.gz 2023-10-21T05:25:59.330 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47897.log 2023-10-21T05:25:59.330 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52759.log.gz 2023-10-21T05:25:59.331 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43422.log 2023-10-21T05:25:59.331 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42393.log.gz 2023-10-21T05:25:59.331 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40156.log 2023-10-21T05:25:59.331 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47897.log.gz 2023-10-21T05:25:59.332 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50754.log 2023-10-21T05:25:59.332 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43422.log.gz 2023-10-21T05:25:59.332 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40267.log 2023-10-21T05:25:59.333 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40156.log.gz 2023-10-21T05:25:59.333 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53054.log 2023-10-21T05:25:59.333 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50754.log.gz 2023-10-21T05:25:59.333 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40861.log 2023-10-21T05:25:59.334 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40267.log.gz 2023-10-21T05:25:59.334 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53992.log 2023-10-21T05:25:59.334 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53054.log.gz 2023-10-21T05:25:59.335 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40392.log 2023-10-21T05:25:59.335 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40861.log.gz 2023-10-21T05:25:59.335 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43816.log 2023-10-21T05:25:59.335 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53992.log.gz 2023-10-21T05:25:59.336 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41094.log 2023-10-21T05:25:59.336 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40392.log.gz 2023-10-21T05:25:59.336 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52947.log 2023-10-21T05:25:59.336 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43816.log.gz 2023-10-21T05:25:59.337 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41185.log 2023-10-21T05:25:59.337 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41094.log.gz 2023-10-21T05:25:59.337 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44820.log 2023-10-21T05:25:59.338 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52947.log.gz 2023-10-21T05:25:59.338 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34633.log 2023-10-21T05:25:59.338 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41185.log.gz 2023-10-21T05:25:59.338 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47928.log 2023-10-21T05:25:59.339 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44820.log.gz 2023-10-21T05:25:59.339 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49142.log 2023-10-21T05:25:59.339 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34633.log.gz 2023-10-21T05:25:59.340 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47928.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.47928.log.gz --verbose 2023-10-21T05:25:59.340 INFO:teuthology.orchestra.run.smithi172.stderr: -- /var/log/ceph/ceph-client.admin.50544.log 2023-10-21T05:25:59.340 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40446.log 2023-10-21T05:25:59.340 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49142.log.gz 2023-10-21T05:25:59.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53509.log 2023-10-21T05:25:59.341 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50544.log.gz 2023-10-21T05:25:59.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53831.log 2023-10-21T05:25:59.342 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40446.log.gz 2023-10-21T05:25:59.342 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51274.log 2023-10-21T05:25:59.342 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53509.log.gz 2023-10-21T05:25:59.343 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45702.log 2023-10-21T05:25:59.343 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53831.log.gz 2023-10-21T05:25:59.343 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46162.log 2023-10-21T05:25:59.343 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51274.log.gz 2023-10-21T05:25:59.344 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46554.log 2023-10-21T05:25:59.344 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45702.log.gz 2023-10-21T05:25:59.344 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43939.log 2023-10-21T05:25:59.345 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46162.log.gz 2023-10-21T05:25:59.345 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49662.log 2023-10-21T05:25:59.345 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46554.log.gz 2023-10-21T05:25:59.345 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45310.log 2023-10-21T05:25:59.346 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43939.log.gz 2023-10-21T05:25:59.346 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44640.log 2023-10-21T05:25:59.346 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49662.log.gz 2023-10-21T05:25:59.347 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54129.log 2023-10-21T05:25:59.347 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45310.log.gz 2023-10-21T05:25:59.347 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40736.log 2023-10-21T05:25:59.347 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44640.log.gz 2023-10-21T05:25:59.348 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48503.log 2023-10-21T05:25:59.348 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.54129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54129.log.gz 2023-10-21T05:25:59.348 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49383.log 2023-10-21T05:25:59.348 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40736.log.gz 2023-10-21T05:25:59.349 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48199.log 2023-10-21T05:25:59.349 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48503.log.gz 2023-10-21T05:25:59.349 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53536.log 2023-10-21T05:25:59.350 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49383.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49383.log.gz 2023-10-21T05:25:59.350 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52733.log 2023-10-21T05:25:59.350 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48199.log.gz 2023-10-21T05:25:59.350 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41402.log 2023-10-21T05:25:59.351 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53536.log.gz 2023-10-21T05:25:59.351 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44151.log 2023-10-21T05:25:59.351 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52733.log.gz 2023-10-21T05:25:59.352 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50239.log 2023-10-21T05:25:59.352 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41402.log.gz 2023-10-21T05:25:59.352 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50300.log 2023-10-21T05:25:59.352 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44151.log.gz 2023-10-21T05:25:59.353 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51150.log 2023-10-21T05:25:59.353 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50239.log.gz 2023-10-21T05:25:59.353 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51426.log 2023-10-21T05:25:59.353 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50300.log.gz 2023-10-21T05:25:59.354 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53590.log 2023-10-21T05:25:59.354 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51150.log.gz 2023-10-21T05:25:59.354 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39250.log 2023-10-21T05:25:59.355 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51426.log.gz 2023-10-21T05:25:59.355 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53106.log 2023-10-21T05:25:59.355 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53590.log.gz 2023-10-21T05:25:59.355 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46891.log 2023-10-21T05:25:59.356 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39250.log.gz 2023-10-21T05:25:59.356 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48229.log 2023-10-21T05:25:59.356 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53106.log.gz 2023-10-21T05:25:59.357 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-21T05:25:59.357 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46891.log.gz 2023-10-21T05:25:59.357 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40969.log 2023-10-21T05:25:59.357 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48229.log.gz 2023-10-21T05:25:59.358 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41352.log 2023-10-21T05:25:59.358 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53375.log 2023-10-21T05:25:59.358 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40969.log.gz 2023-10-21T05:25:59.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41352.log.gz 2023-10-21T05:25:59.359 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51181.log 2023-10-21T05:25:59.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53375.log.gz 2023-10-21T05:25:59.359 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51305.log 2023-10-21T05:25:59.360 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52300.log 2023-10-21T05:25:59.360 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51181.log.gz 2023-10-21T05:25:59.360 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51853.log 2023-10-21T05:25:59.360 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51305.log.gz 2023-10-21T05:25:59.361 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49263.log 2023-10-21T05:25:59.361 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52300.log.gz 2023-10-21T05:25:59.361 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53753.log 2023-10-21T05:25:59.362 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51853.log.gz 2023-10-21T05:25:59.362 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44212.log 2023-10-21T05:25:59.362 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49263.log.gz 2023-10-21T05:25:59.362 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53753.log.gz 2023-10-21T05:25:59.363 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54155.log 2023-10-21T05:25:59.363 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49203.log 2023-10-21T05:25:59.363 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44212.log.gz 2023-10-21T05:25:59.364 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46921.log 2023-10-21T05:25:59.364 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.54155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54155.log.gz 2023-10-21T05:25:59.364 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40321.log 2023-10-21T05:25:59.364 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49203.log.gz 2023-10-21T05:25:59.365 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47109.log 2023-10-21T05:25:59.365 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46921.log.gz 2023-10-21T05:25:59.365 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40429.log 2023-10-21T05:25:59.366 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40321.log.gz 2023-10-21T05:25:59.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48719.log 2023-10-21T05:25:59.366 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47109.log.gz 2023-10-21T05:25:59.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40429.log.gz 2023-10-21T05:25:59.367 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39398.log 2023-10-21T05:25:59.367 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52081.log 2023-10-21T05:25:59.367 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48719.log.gz 2023-10-21T05:25:59.368 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40645.log 2023-10-21T05:25:59.368 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39398.log.gz 2023-10-21T05:25:59.368 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52081.log.gz 2023-10-21T05:25:59.368 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43576.log 2023-10-21T05:25:59.369 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43362.log 2023-10-21T05:25:59.369 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40645.log.gz 2023-10-21T05:25:59.369 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48870.log 2023-10-21T05:25:59.370 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43576.log.gz 2023-10-21T05:25:59.370 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50089.log 2023-10-21T05:25:59.370 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43362.log.gz 2023-10-21T05:25:59.370 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50970.log 2023-10-21T05:25:59.371 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48870.log.gz 2023-10-21T05:25:59.371 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50089.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.51486.log 2023-10-21T05:25:59.371 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50089.log.gz 2023-10-21T05:25:59.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42852.log 2023-10-21T05:25:59.372 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50970.log.gz 2023-10-21T05:25:59.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43543.log 2023-10-21T05:25:59.372 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51486.log.gz 2023-10-21T05:25:59.373 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49752.log 2023-10-21T05:25:59.373 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42852.log.gz 2023-10-21T05:25:59.373 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43543.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.41288.log -- replaced with /var/log/ceph/ceph-client.admin.43543.log.gz 2023-10-21T05:25:59.373 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-21T05:25:59.374 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42615.log 2023-10-21T05:25:59.374 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49752.log.gz 2023-10-21T05:25:59.374 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40304.log 2023-10-21T05:25:59.375 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41288.log.gz 2023-10-21T05:25:59.375 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52411.log 2023-10-21T05:25:59.375 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42615.log.gz 2023-10-21T05:25:59.375 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45216.log 2023-10-21T05:25:59.376 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40304.log.gz 2023-10-21T05:25:59.376 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46254.log 2023-10-21T05:25:59.376 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52411.log.gz 2023-10-21T05:25:59.377 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44760.log 2023-10-21T05:25:59.377 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45216.log.gz 2023-10-21T05:25:59.377 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46254.log.gz 2023-10-21T05:25:59.377 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46735.log 2023-10-21T05:25:59.378 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52375.log 2023-10-21T05:25:59.378 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44760.log.gz 2023-10-21T05:25:59.378 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45551.log 2023-10-21T05:25:59.379 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46735.log.gz 2023-10-21T05:25:59.379 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52375.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.52375.log.gz -5 2023-10-21T05:25:59.379 INFO:teuthology.orchestra.run.smithi172.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46705.log 2023-10-21T05:25:59.379 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42237.log 2023-10-21T05:25:59.380 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45551.log.gz 2023-10-21T05:25:59.380 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49964.log 2023-10-21T05:25:59.380 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46705.log.gz 2023-10-21T05:25:59.380 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42237.log.gz 2023-10-21T05:25:59.381 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46284.log 2023-10-21T05:25:59.381 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51763.log 2023-10-21T05:25:59.381 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49964.log.gz 2023-10-21T05:25:59.382 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45824.log 2023-10-21T05:25:59.382 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46284.log.gz 2023-10-21T05:25:59.382 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44489.log 2023-10-21T05:25:59.382 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51763.log.gz 2023-10-21T05:25:59.383 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45824.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-21T05:25:59.383 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45824.log.gz 2023-10-21T05:25:59.383 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44489.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44489.log.gz 2023-10-21T05:25:59.384 INFO:teuthology.orchestra.run.smithi172.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46675.log 2023-10-21T05:25:59.384 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52155.log 2023-10-21T05:25:59.384 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45091.log 2023-10-21T05:25:59.384 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46675.log.gz 2023-10-21T05:25:59.385 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48380.log 2023-10-21T05:25:59.385 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52155.log.gz 2023-10-21T05:25:59.385 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49934.log 2023-10-21T05:25:59.385 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45091.log.gz 2023-10-21T05:25:59.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45642.log 2023-10-21T05:25:59.386 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48380.log.gz 2023-10-21T05:25:59.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50784.log 2023-10-21T05:25:59.387 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49934.log.gz 2023-10-21T05:25:59.387 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50424.log 2023-10-21T05:25:59.387 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45642.log.gz 2023-10-21T05:25:59.388 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47834.log 2023-10-21T05:25:59.388 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50784.log.gz 2023-10-21T05:25:59.388 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47591.log 2023-10-21T05:25:59.388 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50424.log.gz 2023-10-21T05:25:59.389 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47834.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-21T05:25:59.389 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47834.log.gz 2023-10-21T05:25:59.389 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42939.log 2023-10-21T05:25:59.390 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47591.log.gz 2023-10-21T05:25:59.390 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43482.log 2023-10-21T05:25:59.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.42939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42879.log 2023-10-21T05:25:59.390 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42939.log.gz 2023-10-21T05:25:59.391 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40682.log 2023-10-21T05:25:59.391 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43482.log.gz 2023-10-21T05:25:59.391 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43513.log 2023-10-21T05:25:59.392 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42879.log.gz 2023-10-21T05:25:59.392 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51606.log 2023-10-21T05:25:59.392 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40682.log.gz 2023-10-21T05:25:59.392 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39578.log 2023-10-21T05:25:59.393 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43513.log.gz 2023-10-21T05:25:59.393 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54020.log 2023-10-21T05:25:59.393 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51606.log.gz 2023-10-21T05:25:59.394 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49842.log 2023-10-21T05:25:59.394 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39578.log.gz 2023-10-21T05:25:59.394 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.54020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40699.log 2023-10-21T05:25:59.394 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54020.log.gz 2023-10-21T05:25:59.395 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39026.log 2023-10-21T05:25:59.395 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49842.log.gz 2023-10-21T05:25:59.395 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45612.log 2023-10-21T05:25:59.396 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40699.log.gz 2023-10-21T05:25:59.396 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50724.log 2023-10-21T05:25:59.396 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39026.log.gz 2023-10-21T05:25:59.396 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39406.log 2023-10-21T05:25:59.397 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45612.log.gz 2023-10-21T05:25:59.397 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41320.log 2023-10-21T05:25:59.397 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50724.log.gz 2023-10-21T05:25:59.397 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52336.log 2023-10-21T05:25:59.398 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39406.log.gz 2023-10-21T05:25:59.398 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40662.log 2023-10-21T05:25:59.398 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41320.log.gz 2023-10-21T05:25:59.399 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43210.log 2023-10-21T05:25:59.399 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52336.log.gz 2023-10-21T05:25:59.399 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40662.log.gz 2023-10-21T05:25:59.399 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53644.log 2023-10-21T05:25:59.400 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52393.log 2023-10-21T05:25:59.400 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43210.log.gz 2023-10-21T05:25:59.400 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46951.log 2023-10-21T05:25:59.401 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53644.log.gz 2023-10-21T05:25:59.401 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41918.log 2023-10-21T05:25:59.401 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52393.log.gz 2023-10-21T05:25:59.401 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47169.log 2023-10-21T05:25:59.402 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46951.log.gz 2023-10-21T05:25:59.402 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41918.log.gz 2023-10-21T05:25:59.402 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50514.log 2023-10-21T05:25:59.403 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48657.log 2023-10-21T05:25:59.403 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47169.log.gz 2023-10-21T05:25:59.403 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47289.log 2023-10-21T05:25:59.403 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50514.log.gz 2023-10-21T05:25:59.404 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42756.log 2023-10-21T05:25:59.404 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48657.log.gz 2023-10-21T05:25:59.404 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47289.log.gz 2023-10-21T05:25:59.405 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46007.log 2023-10-21T05:25:59.405 INFO:teuthology.orchestra.run.smithi172.stderr: 88.5% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-21T05:25:59.405 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41240.log 2023-10-21T05:25:59.405 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42756.log.gz 2023-10-21T05:25:59.406 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41131.log 2023-10-21T05:25:59.406 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46007.log.gz 2023-10-21T05:25:59.406 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49540.log 2023-10-21T05:25:59.407 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41240.log.gz 2023-10-21T05:25:59.407 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51637.log 2023-10-21T05:25:59.407 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41131.log.gz 2023-10-21T05:25:59.407 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41222.log 2023-10-21T05:25:59.408 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49540.log.gz 2023-10-21T05:25:59.408 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46374.log 2023-10-21T05:25:59.408 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51637.log.gz 2023-10-21T05:25:59.408 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44670.log 2023-10-21T05:25:59.409 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41222.log.gz 2023-10-21T05:25:59.409 INFO:teuthology.orchestra.run.smithi172.stderr:gzip/var/log/ceph/ceph-client.admin.46374.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49812.log 2023-10-21T05:25:59.409 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46374.log.gz 2023-10-21T05:25:59.410 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53670.log 2023-10-21T05:25:59.410 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44670.log.gz 2023-10-21T05:25:59.410 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39327.log 2023-10-21T05:25:59.410 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49812.log.gz 2023-10-21T05:25:59.411 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50330.log 2023-10-21T05:25:59.411 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53670.log.gz 2023-10-21T05:25:59.411 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47048.log 2023-10-21T05:25:59.411 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39327.log.gz 2023-10-21T05:25:59.412 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41834.log 2023-10-21T05:25:59.412 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50330.log.gz 2023-10-21T05:25:59.412 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48626.log 2023-10-21T05:25:59.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47048.log.gz 2023-10-21T05:25:59.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39248.log 2023-10-21T05:25:59.413 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41834.log.gz 2023-10-21T05:25:59.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34598.log 2023-10-21T05:25:59.414 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48626.log.gz 2023-10-21T05:25:59.414 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42369.log 2023-10-21T05:25:59.414 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39248.log.gz 2023-10-21T05:25:59.415 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53482.log 2023-10-21T05:25:59.415 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34598.log.gz 2023-10-21T05:25:59.415 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2023-10-21T05:25:59.415 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42369.log.gz 2023-10-21T05:25:59.416 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52893.log 2023-10-21T05:25:59.416 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53482.log.gz 2023-10-21T05:25:59.416 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40608.log 2023-10-21T05:25:59.416 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz 2023-10-21T05:25:59.417 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40230.log 2023-10-21T05:25:59.417 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52893.log.gz 2023-10-21T05:25:59.417 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51883.log 2023-10-21T05:25:59.418 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40608.log.gz 2023-10-21T05:25:59.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53857.log 2023-10-21T05:25:59.418 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40230.log.gz 2023-10-21T05:25:59.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52282.log 2023-10-21T05:25:59.419 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51883.log.gz 2023-10-21T05:25:59.419 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52525.log 2023-10-21T05:25:59.419 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53857.log.gz 2023-10-21T05:25:59.420 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40375.log 2023-10-21T05:25:59.420 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52282.log.gz 2023-10-21T05:25:59.420 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41792.log 2023-10-21T05:25:59.420 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52525.log.gz 2023-10-21T05:25:59.421 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44306.log 2023-10-21T05:25:59.421 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40375.log.gz 2023-10-21T05:25:59.421 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34704.log 2023-10-21T05:25:59.421 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41792.log.gz 2023-10-21T05:25:59.422 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39605.log 2023-10-21T05:25:59.422 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44306.log.gz 2023-10-21T05:25:59.422 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34704.log.gz 2023-10-21T05:25:59.423 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42267.log 2023-10-21T05:25:59.423 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48410.log 2023-10-21T05:25:59.423 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39605.log.gz 2023-10-21T05:25:59.423 INFO:teuthology.orchestra.run.smithi172.stderr:gzip/var/log/ceph/ceph-client.admin.42267.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49172.log 2023-10-21T05:25:59.424 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42267.log.gz 2023-10-21T05:25:59.424 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51793.log 2023-10-21T05:25:59.424 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48410.log.gz 2023-10-21T05:25:59.425 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49172.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.49172.log.gz --verbose 2023-10-21T05:25:59.425 INFO:teuthology.orchestra.run.smithi172.stderr: -- /var/log/ceph/ceph-client.admin.43331.log 2023-10-21T05:25:59.425 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47865.log 2023-10-21T05:25:59.425 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51793.log.gz 2023-10-21T05:25:59.426 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45031.log 2023-10-21T05:25:59.426 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43331.log.gz 2023-10-21T05:25:59.426 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45582.log 2023-10-21T05:25:59.427 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47865.log.gz 2023-10-21T05:25:59.427 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44850.log 2023-10-21T05:25:59.427 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45031.log.gz 2023-10-21T05:25:59.427 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52707.log 2023-10-21T05:25:59.428 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45582.log.gz 2023-10-21T05:25:59.428 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48810.log 2023-10-21T05:25:59.428 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44850.log.gz 2023-10-21T05:25:59.429 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42666.log 2023-10-21T05:25:59.429 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52707.log.gz 2023-10-21T05:25:59.429 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52681.log 2023-10-21T05:25:59.429 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48810.log.gz 2023-10-21T05:25:59.430 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39164.log 2023-10-21T05:25:59.430 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42666.log.gz 2023-10-21T05:25:59.430 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45123.log 2023-10-21T05:25:59.431 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52681.log.gz 2023-10-21T05:25:59.431 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44181.log 2023-10-21T05:25:59.431 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39164.log.gz 2023-10-21T05:25:59.431 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46130.log 2023-10-21T05:25:59.432 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45123.log.gz 2023-10-21T05:25:59.432 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49722.log 2023-10-21T05:25:59.432 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44181.log.gz 2023-10-21T05:25:59.432 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45732.log 2023-10-21T05:25:59.433 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46130.log.gz 2023-10-21T05:25:59.433 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41960.log 2023-10-21T05:25:59.433 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49722.log.gz 2023-10-21T05:25:59.433 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48840.log 2023-10-21T05:25:59.434 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45732.log.gz 2023-10-21T05:25:59.434 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44880.log 2023-10-21T05:25:59.434 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41960.log.gz 2023-10-21T05:25:59.435 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49323.log 2023-10-21T05:25:59.435 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48840.log.gz 2023-10-21T05:25:59.435 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39112.log 2023-10-21T05:25:59.435 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44880.log.gz 2023-10-21T05:25:59.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49323.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49323.log.gz -5 2023-10-21T05:25:59.436 INFO:teuthology.orchestra.run.smithi172.stderr: --verbose -- /var/log/ceph/ceph.audit.log 2023-10-21T05:25:59.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46766.log 2023-10-21T05:25:59.437 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39112.log.gz 2023-10-21T05:25:59.437 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51365.log 2023-10-21T05:25:59.437 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40213.log 2023-10-21T05:25:59.437 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46766.log.gz 2023-10-21T05:25:59.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49082.log 2023-10-21T05:25:59.438 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51365.log.gz 2023-10-21T05:25:59.438 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40770.log 2023-10-21T05:25:59.439 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40213.log.gz 2023-10-21T05:25:59.439 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53726.log 2023-10-21T05:25:59.439 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49082.log.gz 2023-10-21T05:25:59.439 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39402.log 2023-10-21T05:25:59.440 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40770.log.gz 2023-10-21T05:25:59.440 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52976.log 2023-10-21T05:25:59.440 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53726.log.gz 2023-10-21T05:25:59.441 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47139.log 2023-10-21T05:25:59.441 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39402.log.gz 2023-10-21T05:25:59.441 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47439.log 2023-10-21T05:25:59.441 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52976.log.gz 2023-10-21T05:25:59.442 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46524.log 2023-10-21T05:25:59.442 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47139.log.gz 2023-10-21T05:25:59.442 INFO:teuthology.orchestra.run.smithi172.stderr:gzip/var/log/ceph/ceph-client.admin.47439.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42002.log 2023-10-21T05:25:59.443 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47439.log.gz 2023-10-21T05:25:59.443 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39320.log 2023-10-21T05:25:59.443 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46524.log.gz 2023-10-21T05:25:59.443 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42002.log.gz 2023-10-21T05:25:59.444 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42420.log 2023-10-21T05:25:59.444 INFO:teuthology.orchestra.run.smithi172.stderr:gzip/var/log/ceph/ceph-client.admin.39320.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53132.log 2023-10-21T05:25:59.444 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39320.log.gz 2023-10-21T05:25:59.444 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53697.log 2023-10-21T05:25:59.445 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42420.log.gz 2023-10-21T05:25:59.445 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50119.log 0.0% 2023-10-21T05:25:59.445 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53132.log.gz 2023-10-21T05:25:59.446 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40932.log 2023-10-21T05:25:59.446 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53697.log.gz 2023-10-21T05:25:59.446 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52466.log 2023-10-21T05:25:59.446 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50119.log.gz 2023-10-21T05:25:59.447 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41876.log 2023-10-21T05:25:59.447 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40932.log.gz 2023-10-21T05:25:59.447 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48596.log 2023-10-21T05:25:59.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52466.log.gz 2023-10-21T05:25:59.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41006.log 2023-10-21T05:25:59.448 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41876.log.gz 2023-10-21T05:25:59.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43240.log 2023-10-21T05:25:59.449 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48596.log.gz 2023-10-21T05:25:59.449 INFO:teuthology.orchestra.run.smithi172.stderr:gzip/var/log/ceph/ceph-client.admin.41006.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51030.log 2023-10-21T05:25:59.449 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41006.log.gz 2023-10-21T05:25:59.449 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51823.log 2023-10-21T05:25:59.450 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43240.log.gz 2023-10-21T05:25:59.450 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41750.log 2023-10-21T05:25:59.450 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51030.log.gz 2023-10-21T05:25:59.451 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41530.log 2023-10-21T05:25:59.451 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51823.log.gz 2023-10-21T05:25:59.451 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45491.log 2023-10-21T05:25:59.451 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41750.log.gz 2023-10-21T05:25:59.452 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50909.log 2023-10-21T05:25:59.452 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51546.log 2023-10-21T05:25:59.452 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45491.log.gz 2023-10-21T05:25:59.453 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45854.log 2023-10-21T05:25:59.453 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50909.log.gz 2023-10-21T05:25:59.453 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44610.log 2023-10-21T05:25:59.453 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51546.log.gz 2023-10-21T05:25:59.454 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44790.log 2023-10-21T05:25:59.454 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45854.log.gz 2023-10-21T05:25:59.454 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52867.log 2023-10-21T05:25:59.455 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44610.log.gz 2023-10-21T05:25:59.455 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47651.log 2023-10-21T05:25:59.455 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44790.log.gz 2023-10-21T05:25:59.455 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50394.log 2023-10-21T05:25:59.456 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52867.log.gz 2023-10-21T05:25:59.456 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45153.log 2023-10-21T05:25:59.456 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47651.log.gz 2023-10-21T05:25:59.456 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50394.log: -- /var/log/ceph/ceph-client.admin.42177.log 2023-10-21T05:25:59.457 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50394.log.gz 2023-10-21T05:25:59.457 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40338.log 2023-10-21T05:25:59.457 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45153.log.gz 2023-10-21T05:25:59.458 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52785.log 2023-10-21T05:25:59.458 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42177.log.gz 2023-10-21T05:25:59.458 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43756.log 2023-10-21T05:25:59.458 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40338.log.gz 2023-10-21T05:25:59.459 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52785.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52785.log.gz -5 2023-10-21T05:25:59.459 INFO:teuthology.orchestra.run.smithi172.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47988.log 2023-10-21T05:25:59.459 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43846.log 2023-10-21T05:25:59.460 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43756.log.gz 2023-10-21T05:25:59.460 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40952.log 2023-10-21T05:25:59.460 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47988.log.gz 2023-10-21T05:25:59.460 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43846.log.gz 2023-10-21T05:25:59.461 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2023-10-21T05:25:59.461 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46494.log 2023-10-21T05:25:59.461 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40952.log.gz 2023-10-21T05:25:59.461 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48048.log 2023-10-21T05:25:59.462 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2023-10-21T05:25:59.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39476.log 2023-10-21T05:25:59.462 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46494.log.gz 2023-10-21T05:25:59.463 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53189.log 2023-10-21T05:25:59.463 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48048.log.gz 2023-10-21T05:25:59.463 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51516.log 2023-10-21T05:25:59.463 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39476.log.gz 2023-10-21T05:25:59.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49571.log 2023-10-21T05:25:59.464 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53189.log.gz 2023-10-21T05:25:59.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51516.log.gz 2023-10-21T05:25:59.465 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41077.log 2023-10-21T05:25:59.465 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43636.log 2023-10-21T05:25:59.465 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41530.log.gz 2023-10-21T05:25:59.465 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49571.log.gz 2023-10-21T05:25:59.466 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50879.log 2023-10-21T05:25:59.466 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41077.log.gz 2023-10-21T05:25:59.466 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54072.log 2023-10-21T05:25:59.467 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43636.log.gz 2023-10-21T05:25:59.467 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47560.log 2023-10-21T05:25:59.467 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50879.log.gz 2023-10-21T05:25:59.467 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.54072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49995.log 2023-10-21T05:25:59.468 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54072.log.gz 2023-10-21T05:25:59.468 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40591.log 2023-10-21T05:25:59.468 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47560.log.gz 2023-10-21T05:25:59.469 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52264.log 2023-10-21T05:25:59.469 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49995.log.gz 2023-10-21T05:25:59.469 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53430.log 2023-10-21T05:25:59.469 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40591.log.gz 2023-10-21T05:25:59.470 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47199.log 2023-10-21T05:25:59.470 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52264.log.gz 2023-10-21T05:25:59.470 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48109.log 2023-10-21T05:25:59.473 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53430.log.gz 2023-10-21T05:25:59.473 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2023-10-21T05:25:59.474 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47199.log.gz 2023-10-21T05:25:59.474 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51000.log 2023-10-21T05:25:59.474 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48109.log.gz 2023-10-21T05:25:59.474 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52117.log 2023-10-21T05:25:59.475 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2023-10-21T05:25:59.475 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53218.log 2023-10-21T05:25:59.475 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51000.log.gz 2023-10-21T05:25:59.476 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52117.log.gz 2023-10-21T05:25:59.476 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43696.log 2023-10-21T05:25:59.476 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41623.log 2023-10-21T05:25:59.476 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53218.log.gz 2023-10-21T05:25:59.477 INFO:teuthology.orchestra.run.smithi172.stderr:gzip/var/log/ceph/ceph-client.admin.43696.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46645.log 2023-10-21T05:25:59.477 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43696.log.gz 2023-10-21T05:25:59.477 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48931.log 2023-10-21T05:25:59.477 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41623.log.gz 2023-10-21T05:25:59.478 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39247.log 2023-10-21T05:25:59.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46645.log.gz 2023-10-21T05:25:59.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53160.log 2023-10-21T05:25:59.479 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48931.log.gz 2023-10-21T05:25:59.479 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44910.log 2023-10-21T05:25:59.479 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39247.log.gz 2023-10-21T05:25:59.479 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5/var/log/ceph/ceph-client.admin.53160.log: --verbose -- /var/log/ceph/ceph-client.admin.49782.log 2023-10-21T05:25:59.480 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53160.log.gz 2023-10-21T05:25:59.480 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45461.log 2023-10-21T05:25:59.480 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44910.log.gz 2023-10-21T05:25:59.481 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47079.log 2023-10-21T05:25:59.481 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49782.log.gz 2023-10-21T05:25:59.481 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40716.log 2023-10-21T05:25:59.481 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45461.log.gz 2023-10-21T05:25:59.482 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52577.log 2023-10-21T05:25:59.482 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47079.log.gz 2023-10-21T05:25:59.482 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53805.log 2023-10-21T05:25:59.483 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40716.log.gz 2023-10-21T05:25:59.483 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39138.log 2023-10-21T05:25:59.483 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52577.log.gz 2023-10-21T05:25:59.483 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44120.log 2023-10-21T05:25:59.484 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53805.log.gz 2023-10-21T05:25:59.484 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.39138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39138.log.gz 2023-10-21T05:25:59.484 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53349.log 2023-10-21T05:25:59.485 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40807.log 2023-10-21T05:25:59.485 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44120.log.gz 2023-10-21T05:25:59.485 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42126.log 2023-10-21T05:25:59.485 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53349.log.gz 2023-10-21T05:25:59.486 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40807.log.gzgzip 2023-10-21T05:25:59.486 INFO:teuthology.orchestra.run.smithi172.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51973.log 2023-10-21T05:25:59.486 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51395.log 2023-10-21T05:25:59.487 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42126.log.gz 2023-10-21T05:25:59.487 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47530.log 2023-10-21T05:25:59.487 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51973.log.gz 2023-10-21T05:25:59.487 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41114.log 2023-10-21T05:25:59.488 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51395.log.gz 2023-10-21T05:25:59.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51090.log 2023-10-21T05:25:59.488 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47530.log.gz 2023-10-21T05:25:59.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49233.log 2023-10-21T05:25:59.489 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41114.log.gz 2023-10-21T05:25:59.489 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52629.log 2023-10-21T05:25:59.489 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51090.log.gz 2023-10-21T05:25:59.490 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51943.log 2023-10-21T05:25:59.490 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49233.log.gz 2023-10-21T05:25:59.490 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52629.log: -- /var/log/ceph/ceph-client.admin.43999.log 2023-10-21T05:25:59.490 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52629.log.gz 2023-10-21T05:25:59.491 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41572.log 2023-10-21T05:25:59.491 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51943.log.gz 2023-10-21T05:25:59.491 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46464.log 2023-10-21T05:25:59.492 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43999.log.gz 2023-10-21T05:25:59.492 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51913.log 2023-10-21T05:25:59.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41572.log: /var/log/ceph/ceph-client.admin.46464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47743.log 2023-10-21T05:25:59.492 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46464.log.gz 2023-10-21T05:25:59.493 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45762.log 2023-10-21T05:25:59.493 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51913.log.gz 2023-10-21T05:25:59.493 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40250.log 2023-10-21T05:25:59.493 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47743.log.gz 2023-10-21T05:25:59.494 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40412.log 2023-10-21T05:25:59.494 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45762.log.gz 2023-10-21T05:25:59.494 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52603.log 2023-10-21T05:25:59.495 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40250.log.gz 2023-10-21T05:25:59.495 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49631.log 2023-10-21T05:25:59.495 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40412.log.gz 2023-10-21T05:25:59.496 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50026.log 2023-10-21T05:25:59.496 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52603.log.gz 2023-10-21T05:25:59.496 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46037.log 2023-10-21T05:25:59.496 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49631.log.gz 2023-10-21T05:25:59.497 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45340.log 2023-10-21T05:25:59.497 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50026.log.gz 2023-10-21T05:25:59.497 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41040.log 2023-10-21T05:25:59.498 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46037.log.gz 2023-10-21T05:25:59.498 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52063.log 2023-10-21T05:25:59.498 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45340.log.gz 2023-10-21T05:25:59.498 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52813.log 2023-10-21T05:25:59.499 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41040.log.gz 2023-10-21T05:25:59.499 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46404.log 2023-10-21T05:25:59.499 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52063.log.gz 2023-10-21T05:25:59.500 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42800.log 2023-10-21T05:25:59.500 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52813.log.gz 2023-10-21T05:25:59.500 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.46404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43180.log 2023-10-21T05:25:59.500 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46404.log.gz 2023-10-21T05:25:59.501 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47379.log 2023-10-21T05:25:59.501 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42800.log.gz 2023-10-21T05:25:59.501 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41708.log 2023-10-21T05:25:59.502 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43180.log.gz 2023-10-21T05:25:59.502 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51576.log 2023-10-21T05:25:59.502 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47379.log.gz 2023-10-21T05:25:59.502 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43786.log 2023-10-21T05:25:59.503 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41708.log.gz 2023-10-21T05:25:59.503 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53779.log 2023-10-21T05:25:59.503 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51576.log.gz 2023-10-21T05:25:59.504 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52136.log 2023-10-21T05:25:59.504 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43786.log.gz 2023-10-21T05:25:59.504 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43452.log 2023-10-21T05:25:59.504 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53779.log.gz 2023-10-21T05:25:59.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45521.log 2023-10-21T05:25:59.505 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52136.log.gz 2023-10-21T05:25:59.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.43452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41060.log 2023-10-21T05:25:59.505 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43452.log.gz 2023-10-21T05:25:59.506 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.45521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47469.log 2023-10-21T05:25:59.506 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45521.log.gz 2023-10-21T05:25:59.506 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42726.log 2023-10-21T05:25:59.507 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41060.log.gz 2023-10-21T05:25:59.507 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.47469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42480.log 2023-10-21T05:25:59.507 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47469.log.gz 2023-10-21T05:25:59.507 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44059.log 2023-10-21T05:25:59.508 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42726.log.gz 2023-10-21T05:25:59.508 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42044.log 2023-10-21T05:25:59.508 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42480.log.gz 2023-10-21T05:25:59.508 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48533.log 2023-10-21T05:25:59.509 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44059.log.gz 2023-10-21T05:25:59.509 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53002.log 2023-10-21T05:25:59.509 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42044.log.gz 2023-10-21T05:25:59.510 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40139.log 2023-10-21T05:25:59.510 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48533.log.gz 2023-10-21T05:25:59.510 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44089.log/var/log/ceph/ceph-client.admin.53002.log: 2023-10-21T05:25:59.510 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53002.log.gz 2023-10-21T05:25:59.511 INFO:teuthology.orchestra.run.smithi172.stderr:gzip/var/log/ceph/ceph-client.admin.40139.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53563.log 2023-10-21T05:25:59.511 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40139.log.gz 2023-10-21T05:25:59.511 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44089.log.gz 2023-10-21T05:25:59.512 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53080.log 2023-10-21T05:25:59.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53886.log 2023-10-21T05:25:59.512 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53563.log.gz 2023-10-21T05:25:59.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51456.log 2023-10-21T05:25:59.513 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53080.log.gz 2023-10-21T05:25:59.513 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.53886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53886.log.gz 2023-10-21T05:25:59.513 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50484.log 2023-10-21T05:25:59.513 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44519.log 2023-10-21T05:25:59.514 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51456.log.gz 2023-10-21T05:25:59.514 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.50484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49293.log 2023-10-21T05:25:59.514 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50484.log.gz 2023-10-21T05:25:59.515 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44519.log.gz 2023-10-21T05:25:59.515 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41488.log 2023-10-21T05:25:59.515 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52484.log 2023-10-21T05:25:59.515 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.49293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49293.log.gz 2023-10-21T05:25:59.516 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42828.log 2023-10-21T05:25:59.516 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41488.log.gz 2023-10-21T05:25:59.516 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51120.log 2023-10-21T05:25:59.517 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52484.log.gz 2023-10-21T05:25:59.517 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41202.log 2023-10-21T05:25:59.517 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.42828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42828.log.gz 2023-10-21T05:25:59.517 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.51120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40824.log 2023-10-21T05:25:59.518 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51120.log.gz 2023-10-21T05:25:59.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.41202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48991.log 2023-10-21T05:25:59.518 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41202.log.gz 2023-10-21T05:25:59.518 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52920.log 2023-10-21T05:25:59.519 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.40824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40824.log.gz 2023-10-21T05:25:59.519 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.48991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44242.log 2023-10-21T05:25:59.519 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48991.log.gz 2023-10-21T05:25:59.520 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.52920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52920.log.gz 2023-10-21T05:25:59.520 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.44242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44242.log.gz 2023-10-21T05:25:59.520 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-21T05:25:59.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-21T05:25:59.521 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41572.log.gz 2023-10-21T05:26:08.738 INFO:teuthology.orchestra.run.smithi172.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-21T05:26:22.977 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-21T05:59:23.020 INFO:teuthology.orchestra.run.smithi172.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-21T06:14:51.233 INFO:teuthology.orchestra.run.smithi172.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-21T06:15:05.632 INFO:teuthology.orchestra.run.smithi172.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-21T06:15:05.633 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-21T06:15:05.633 INFO:teuthology.orchestra.run.smithi172.stderr:real 49m6.618s 2023-10-21T06:15:05.633 INFO:teuthology.orchestra.run.smithi172.stderr:user 46m58.755s 2023-10-21T06:15:05.634 INFO:teuthology.orchestra.run.smithi172.stderr:sys 2m24.251s 2023-10-21T06:15:05.635 INFO:tasks.ceph:Archiving logs... 2023-10-21T06:15:05.636 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433369/remote/smithi172/log 2023-10-21T06:15:05.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-21T06:17:19.694 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-21T06:17:19.710 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-21T06:17:19.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-21T06:17:19.780 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-21T06:17:19.792 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-21T06:17:19.793 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-21T06:17:19.883 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2023-10-21T06:17:19.884 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2023-10-21T06:17:19.884 INFO:teuthology.orchestra.run.smithi172.stdout:*hv01.front.sepi 44.4.53.2 2 u 228 256 377 0.133 -2.217 0.081 2023-10-21T06:17:19.885 INFO:teuthology.orchestra.run.smithi172.stdout:+hv02.front.sepi 63.231.80.2 3 u 248 256 377 0.177 +0.643 0.350 2023-10-21T06:17:19.885 INFO:teuthology.orchestra.run.smithi172.stdout:+hv03.front.sepi 74.6.168.72 3 u 48 256 377 0.173 -2.044 0.173 2023-10-21T06:17:19.885 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-21T06:17:19.887 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-21T06:17:19.896 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-21T06:17:19.896 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-21T06:17:19.908 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-21T06:17:19.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-21T06:17:19.930 INFO:teuthology.task.internal:Duration was 8526.871152 seconds 2023-10-21T06:17:19.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-21T06:17:19.941 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-21T06:17:19.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-21T06:17:20.166 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-21T06:17:20.167 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2023-10-21T06:17:20.167 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-21T06:17:20.229 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-21T06:17:20.230 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-21T06:17:20.292 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-21T06:17:20.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-21T06:17:20.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-21T06:17:20.626 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-21T06:17:20.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-21T06:17:20.687 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-21T06:17:20.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-21T06:17:20.722 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2023-10-21T06:17:20.757 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-21T06:17:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T06:17:20.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-21T06:17:20.816 INFO:teuthology.task.internal:Transferring archived files... 2023-10-21T06:17:20.818 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433369/remote/smithi172 2023-10-21T06:17:20.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-21T06:17:20.868 INFO:teuthology.task.internal:Removing archive directory... 2023-10-21T06:17:20.868 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-21T06:17:20.914 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-21T06:17:20.927 INFO:teuthology.task.internal:Not uploading archives. 2023-10-21T06:17:20.928 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-21T06:17:20.940 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-21T06:17:20.941 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-21T06:17:20.979 INFO:teuthology.orchestra.run.smithi172.stdout: 655476 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 21 06:17 /home/ubuntu/cephtest 2023-10-21T06:17:20.980 INFO:teuthology.orchestra.run.smithi172.stdout: 793388 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 21 04:05 /home/ubuntu/cephtest/mnt.0 2023-10-21T06:17:20.980 INFO:teuthology.orchestra.run.smithi172.stdout: 793774 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 21 04:06 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-21T06:17:20.980 INFO:teuthology.orchestra.run.smithi172.stdout: 807680 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 21 04:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-21T06:17:20.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T06:17:20.982 INFO:teuthology.orchestra.run.smithi172.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-21T06:17:20.982 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi172 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-21T06:17:20.990 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-21T06:17:21.001 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-21T06:17:21.015 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi172 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-21T06:17:21.016 INFO:teuthology.nuke:Checking targets against current locks 2023-10-21T06:17:21.044 DEBUG:teuthology.nuke:shortname: smithi172 2023-10-21T06:17:21.045 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-21T06:17:21.069 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433369', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-21 03:45:45.604199', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPC+zfLfb5HR102ZeiDb68RvA1E5FrxGqtkVP/oLWYY5xf6lz9rSaD7Wy5qIbUtrDsS7z0KZ/yJdQAhgMzPfMtY='} 2023-10-21T06:17:21.116 INFO:teuthology.orchestra.console.smithi172:Power off 2023-10-21T06:17:21.117 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-21T06:17:21.139 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2023-10-21T06:17:21.140 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:17:21.163 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-21T06:17:25.164 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:17:25.274 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-21T06:17:29.278 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:17:29.388 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-21T06:17:33.390 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:17:33.505 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2023-10-21T06:17:33.607 INFO:teuthology.orchestra.console.smithi172:Power off completed 2023-10-21T06:17:33.834 INFO:teuthology.run:Summary data: description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/ec features/layering msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 8526.87115240097 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi172 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_rraja@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=25638d60a1044fe1a8f0842d6f95c1f0 status: fail success: false 2023-10-21T06:17:33.835 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T06:17:33.938 INFO:teuthology.run:FAIL